Looking back, Application Security has evolved significantly in the last couple of decades. In the early 2000s, SQL injection and Cross Site Scripting (XSS) attacks were a nightmare for cybersecurity teams as attackers easily bypassed network firewalls through attacks at the application layer. Since traditional network firewalls at that time were not application-aware, these attacks proved a blind spot allowing attackers to compromise web applications easily. Hence, the computer industry developed countermeasures which included and not limited to web application firewalls (WAF), source code security reviews, and DevSecOps, who automate these checks within CI/CD pipelines to and allow security at speed with dynamic application security testing (DAST) and static application security testing (SAST) solutions to improve an environment’s security posture.

In today’s complex Software Landscape, AI (Artificial Intelligence) and ML (Machine Learning) enabled Apps are becoming the norm (de rigueur) since these are perceived to provide users with a great UX (User Experience) and ease of use. These Systems are being adopted across the world by Enterprises and Government world-wide, as machine learning algorithms, boost an application’s capability by using models that are trained on data to build up their decision-making abilities, similar to how a human being learns from trial and error. In basic terms, the more data a machine learning model is trained on, the more accurate it becomes. Once deemed fit for production, these models are placed behind applications that typically expose public APIs, which can be queried for results.

However, the adoption of these applications in sensitive industries like travel, banking, hospitality, medicine, and finance, along with their access to sensitive training data, opens new Attack Surfaces for Bad Actors to exploit. Consequently, new techniques of attacks are being developed by these bad actors to target the workings of machine learning applications today similar to the “sophisticated attacks” of earlier decades like SQL injections. Cybersecurity teams, typically assess applications via traditional security processes, such as hardening, patching, vulnerability assessments which are done at the infrastructure and application levels.

While this is all good practice, these techniques processes do not address cybercriminal’s primary objective of manipulating the way AI/ML decisions making process i.e., the algorithm with which AI and machine learning applications reach decisions. Bad Actors accomplish this by trying to ascertain how the model works so they can reverse engineer it for further attacks, determine what data the model was trained on, along with revealing sensitive characteristics to interfere with the workings of AI applications and make them reach the wrong decisions. It is reported that these attacks are growing in number and have been successfully exploited in production-based AI Applications using some specific well-known attack strategies such as Membership Inference Attack, Data Poisoning Attack, Model Evasion Attack enumerated below:

Membership Inference Attack
During inference attacks on AI applications, an attacker tries to ascertain the inner workings of a model with what type of data was used to train it by using APIs exposed by ML models which may provide responses as confidence scores and give stronger scores if the data they are fed matches the data they were trained on. With access to this API, the attacker can run different queries and analyze the machine learning model responses resulting in the AI model disclosing highly sensitive data.

Poisoning attacks
In poisoning attacks on AI applications, the bad actor “pollutes” the data on which the model is being trained to confuse its decision-making process. Generally, most companies again do nocreate training data from scratch and often leverage pre-built training data to run through their machine learning models. If an attacker can compromise this data repository via a security vulnerability and inject his own data into this store, then the machine learning model will be trained to accept a malicious input right from the start – for instance, if an attacker can “corrupt” the data of a self-driving car software by indicating that a “Stop Sign” is to speed up and not stop, they can make this car get into accidents at stop signs! Attackers usually bide their time and wait for the data store to reach a certain level of market acceptance before trying to “poison” them. Model training is also not a one-time activity – data store might perfect in the beginning and could be polluted by an attacker later once the bad actor is confident that this will not be discovered.

Evasion attacks
In evasion attacks on AI systems, the attacker will try to trick the model by providing subtly changed data, that are not noticeable to a human, resulting in dramatically different decisions being made by machine learning models. This data type is also known as an adversarial sample and can trick AI-powered systems – For instance, using a self-driving car as an example, simply putting pieces of tape onto a stop sign can result in a machine learning model not recognizing it causing car accidents.

 

Attacks on AI-based Systems are becoming more common-places and cybersecurity teams need to understand this new breed of attacks as traditional cybersecurity controls will not protect against these vulnerabilities, and new types of controls need to be put in place. Since there is no quick patch to address these issues, some countermeasures to harden such AI/ML based Applications are suggested here – please note that this is not inclusive but just some basic strategies for consideration:

1. Threat modeling of machine learning models should be carried out and made a standard practice before choosing any new or pre-built model.
2. Models should be hardened to sanitize confidence scores in their responses with developers possibly being provided full confidence score details while end users are provided only a summarized score. This would greatly increase the difficulty for an attacker to assess the underlying logic or data that the model was trained on.
3. Detection controls need to be updated to alert if a particular attacker is repeatedly querying a particular machine learning API which could point to a possible inference attack.
4. Machine learning models should be trained on adversarial samples to make them resilience to such attacks using such adversarial data.
5. Data stores to train machine learning models must undergo rigorous security testing to remove vulnerabilities that may allow attacks to gain access to their data and poison them.
6. Periodic testing of Data Stores must be performed to ensure that they have not been “poisoned” subsequently.
7. Using Public or open-source data for training should be kept to a minimum – they may be easier to use for training models, but these can be compromised which could lead to the model being corrupted.

Hence, it is worth noting that that AI attacks are only poised to increase with time and cybersecurity teams need to take a proactive approach to ensure that proper technical controls and governance are implemented to keep these applications running smoothly.