Adversarial machine learning: attacks, examples and defenses

What is adversarial machine learning?

Adversarial machine learning is the study of attacks that cheat or mislead a model with malicious input, and of the ways to stop them. Attackers use these techniques to crash or steer a system; researchers and red teams use the same techniques to find weaknesses before attackers do. An attack crafted for one model often works on other models trained on different datasets or with different architectures.

Attacks are divided by what the attacker knows. In a white-box attack, the attacker can see the inner workings of the model, including its weights and gradients. In a black-box attack, the attacker sees only the model’s outputs and probes it with queries. Either way, an attacker who manipulates the inputs can make a deployed system misread them, for example misidentify stop signs on a real road.

The attacks apply to all three main types of machine learning: supervised learning (classifiers such as spam filters and image recognition), unsupervised learning (clustering and anomaly detection) and reinforcement learning (agents that learn by trial and error).

A short history: from spam filters to the panda

The problem is older than deep learning. In 2004, researchers showed that linear spam filters could be evaded by inserting “good words” into a message, and that one learned filter could be beaten by another that picked those words automatically. In 2012 and 2013, Battista Biggio and colleagues demonstrated the first gradient-based attacks on support vector machines and neural networks.

In December 2013, Christian Szegedy and colleagues, among them Ian Goodfellow, showed that deep image classifiers can be fooled by perturbations a person cannot see, and that the same perturbation often fools a second network trained on different data. A year later, Goodfellow, Jonathon Shlens and Szegedy published the fast gradient sign method (FGSM) and the picture that became the emblem of the field: a panda classified as “panda” with 57.7% confidence and, after a layer of noise too faint to notice, as “gibbon” with 99.3% confidence.

Their explanation still holds: the cause is not overfitting but the largely linear behavior of neural networks in high dimensions. Thousands of tiny changes, one per pixel, add up to a large change in the output.

Examples of adversarial attacks in the real world

What is an example of adversarial AI outside the lab? Four cases are cited most often:

• Stop signs: in 2017, Kevin Eykholt and colleagues put black and white stickers on a stop sign, and a road-sign classifier read it as “Speed Limit 45” in 100% of lab images and in 84.8% of video frames taken from a moving car.
• A turtle that is a rifle: in 2017, Anish Athalye and colleagues at MIT 3D-printed a turtle whose texture made an ImageNet classifier consistently label it a rifle.
• Lane markings: in 2019, Tencent Keen Security Lab placed small stickers on a road that made Tesla Autopilot’s lane recognition steer the car into the opposite lane. Tesla replied that the test changed the physical environment and that drivers can override Autopilot at any time.
• A poisoned chatbot: in 2016, internet trolls fed Microsoft’s Tay chatbot offensive tweets until it repeated them, and Microsoft shut it down after 16 hours.

Types of adversarial attacks on machine learning

AML attacks fall into four groups, depending on when they strike and what they want:

• Evasion, or misclassification input, is the most common variant. The attacker changes an input at inference time, such as an image, a file or an email, so that the deployed model misclassifies it. Malware and spam that slip past filters are evasion attacks. They can be untargeted (any wrong answer) or targeted (a chosen answer, such as “Speed Limit 45”).
• Data poisoning occurs when an attacker places inaccurate or malicious data in a training set, making the output less accurate or steering it. The aim is to degrade the learning process and the usefulness of the model. A backdoor, or Trojan, is planted during training and triggered after deployment: the model behaves normally until it sees a specific pattern.
• Model extraction: the attacker queries a model and uses the answers to build a copy. In 2016, Florian Tramèr and colleagues extracted models from the BigML and Amazon Machine Learning prediction services with near-perfect fidelity.
• Privacy attacks: membership inference tells whether a given record was in the training data, and model inversion reconstructs training data, such as approximate face images, from a model’s outputs.

How attackers craft adversarial examples

Most evasion attacks are optimization problems: find the smallest change to the input that flips the model’s decision. Three methods are the standard reference points:

• FGSM takes one step. It computes the gradient of the model’s loss with respect to the input and nudges every pixel by a tiny amount in the direction that increases the loss. It is fast and relatively weak.
• Projected gradient descent (PGD), formalized by Aleksander Madry and colleagues in 2017, repeats small steps and projects each result back within an allowed distance from the original. It is the usual benchmark attack for testing how well a model resists.
• The Carlini and Wagner attack (2016) searches for the smallest change that forces a chosen wrong answer. It is slower but much stronger, and it was built to test defenses.

A black-box attacker without access to gradients can estimate them from many queries, or build adversarial examples on a substitute model and rely on transfer to the real one.

Adversarial attacks on large language models

Since ChatGPT arrived in late 2022, much of the adversarial work has moved to plain text. Is ChatGPT a GAN? No. It is a large language model built on the transformer architecture, trained to predict the next word and then tuned with reinforcement learning from human feedback, as OpenAI described for InstructGPT. The “adversarial” in generative adversarial networks refers to two networks training against each other, not to attacks. Language models are attacked with the same logic as image classifiers, though.

Prompt injection is first on the OWASP Top 10 for LLM Applications 2025. In a direct injection, a user types instructions that override the system’s rules. In an indirect injection, described by Kai Greshake and colleagues in February 2023, the instructions hide in a web page, an email or a file that the model reads; they demonstrated it against Bing’s GPT-4 chat. For an AI agent that can send email or call other systems, a hidden instruction works like remote code execution.

Jailbreaks can be automated. In July 2023, Andy Zou and colleagues generated adversarial suffixes, strings of odd tokens appended to a request, by gradient search on open models. Suffixes found on Vicuna transferred to the public versions of ChatGPT, Bard and Claude.

Poisoning is cheaper than expected. In October 2025, Anthropic, the UK AI Security Institute and the Alan Turing Institute showed that about 250 poisoned documents were enough to plant a backdoor in every model they trained, from 600 million to 13 billion parameters, regardless of size. For the largest model, that was 0.00016% of its training tokens.

Defense against AML

In 2026 there is still no specific defense that stops every attack, but several techniques make attacks harder. Traditional ways of making models more reliable, such as weight decay or dropout, generally do not provide a practical defense on their own. Two methods dominated early research:

• Adversarial training is a brute force solution: generate many adversarial examples and train the model explicitly not to be fooled by them. Training on PGD examples, as Madry and colleagues proposed, remains the standard empirical defense. It costs extra compute, often lowers accuracy on clean inputs, and needs constant maintenance, because it protects mainly against the attacks it has seen.
• Defensive distillation teaches a model to output the probabilities of different classes rather than hard decisions. The probabilities come from an earlier model trained for the same task on hard class labels. This smooths the model’s surface in the directions an attacker usually exploits. In 2016 its authors reported that it cut attack success from about 95% to 0.5%, but within months Carlini and Wagner built attacks that succeeded 100% of the time against distilled networks.

Distillation has one model learn from the outputs of another, previously trained model. That two-model setup resembles generative adversarial networks (GANs), in which two neural networks train together, but in a GAN they compete, while in distillation one teaches the other.

A warning came in 2018. Anish Athalye, Nicholas Carlini and David Wagner examined nine defenses from the ICLR 2018 conference that claimed white-box security: seven relied on obfuscated gradients, which hide the gradient from the attacker without making the model any harder to fool, and their adaptive attacks fully broke six and partly broke a seventh. A defense counts only if it holds against an attacker who knows how it works.

Certified defenses give a mathematical guarantee instead of a test result. Randomized smoothing, published by Jeremy Cohen and colleagues in 2019, certifies that a prediction cannot change within a small radius around the input; on ImageNet it reached 49% certified accuracy for small perturbations, well below normal accuracy. In practice, teams combine model-level methods with operational controls:

• Input validation and transformation, such as resizing, compression and noise filtering, plus monitoring for unusual inputs and drops in confidence.
• Data provenance and sanitization against poisoning: know where training data comes from and check it before each training run.
• Rate limits, query monitoring and minimal outputs against extraction (Tramèr’s team found that hiding confidence scores alone does not stop it).
• For LLM applications: least privilege for tools, human approval for high-risk actions, and untrusted content kept apart from instructions, as OWASP recommends.
• Regular red teaming, using a shared vocabulary such as NIST’s taxonomy of attacks and mitigations (AI 100-2 E2025, March 2025).

Why defend against adversarial examples, and why it is hard

When we think about AI security research, we usually think about the toughest issues in the field: how can we make sure that reinforcement learning agents much smarter than humans behave as their designers intended?

Adversarial examples show that even simple, modern algorithms, for both supervised learning and reinforcement learning, can already behave in surprising ways that we do not intend.

They are hard to defend against because it is hard to build a theoretical model of how they are created. They are solutions to an optimization problem that is non-linear and non-convex for many ML models, including neural networks. Without good theoretical tools to describe those solutions, it is very hard to argue that a defense rules out a whole set of attacks. Defense is also hard because it requires models to produce good results for every possible input. Most of the time, models work very well, but only on a very small share of all the inputs they may encounter. An important area of research is designing defenses that hold against powerful, adaptive attackers.

This is something to keep in mind when deploying machine learning models. Rather than blindly trusting the models to deliver correct results, we must guard against these attacks and think twice before accepting their decisions. The gap between what designers intend and how algorithms behave is exactly what this field tries to close.