Machine learning models are not neutral tools. They are systems with attack surfaces, and those surfaces are being actively exploited. Adversarial machine learning is the discipline that studies how attackers manipulate, evade, and compromise ML systems, and how defenders protect them.
For cyber defenders in 2026, this is no longer a research topic. It is operational reality. The ML-based detection tools in your SOC, the AI models processing threat intelligence, the LLM-powered automation in your security workflow: all of them are targets. Understanding how adversarial attacks against ML systems work is what makes you able to defend against them.
What Is Adversarial Machine Learning?
Adversarial machine learning (AML) is the study of attacks against machine learning systems and the defences built to counter them. NIST documents four main adversarial ML attack categories: evasion, poisoning, privacy, and abuse attacks. Each targets a different phase of the ML lifecycle and requires different defensive controls.
The term "adversarial" refers to the adversary-centric framing: rather than asking "how does this model perform on clean data?", adversarial ML asks "how does this model perform when an intelligent attacker is deliberately trying to make it fail?"
For defenders, the practical implication is this: if your organisation depends on ML-based tools for detection, classification, or decision-making, those tools have attack surfaces that traditional security controls were not designed to address. MITRE ATLAS (Adversarial Threat Landscape for Artificial Intelligence Systems) is the definitive framework for understanding and defending against these threats. As of February 2026, ATLAS v5.4.0 contains 16 tactics, 84 techniques, and 42 real-world case studies - growing rapidly as the attack surface expands.
What Are the Four Main Adversarial ML Attack Categories?
Evasion Attacks
Evasion attacks occur at inference time. The attacker crafts inputs designed to cause the model to produce incorrect outputs - without changing the underlying content in a way that a human would detect.
The canonical example is adversarial examples against image classifiers: imperceptible pixel-level perturbations that cause a correctly-classified image to be misclassified with high confidence. For cyber defenders, the more directly relevant application is malware evasion: attackers modify malware samples to evade ML-based detection while preserving full malicious functionality. A sample that bypasses your ML detector does not need to exploit a code vulnerability in the detector. It just needs to fall outside the decision boundary the model learned during training.
For LLM-based systems, evasion includes guardrail bypass through encoding tricks, Unicode manipulation, and carefully constructed prompts that cause a model to produce outputs its safety controls were designed to prevent.
MITRE ATLAS technique: AML.T0015 (Evade ML Model)
Defender implication: ML-based detectors need adversarial robustness testing as part of their validation. A detector that performs well on clean samples but fails on adversarially perturbed ones provides a false sense of security.
Poisoning Attacks
Poisoning attacks occur at training time. Adversaries inject inaccurate or malicious data into training datasets - sometimes controlling only a few dozen training samples - to influence the model's learned behaviour. The attack can be designed to degrade overall model performance (availability attack) or to embed a backdoor: a specific trigger that causes the model to misclassify a particular input in a way the attacker controls (integrity attack).
For defenders operating threat intelligence systems or any model that learns from data sources partially outside your control, poisoning is a realistic threat. An attacker who can influence what goes into your training data can influence what your model learns to detect - and what it learns to ignore.
Data poisoning and backdoor attacks are among the fastest-growing documented attack categories in ATLAS. The Spring 2025 ATLAS release added RAG Poisoning and False RAG Entry Injection as specific techniques, reflecting the rapid adoption of retrieval-augmented generation in enterprise AI deployments.
MITRE ATLAS technique: AML.T0020 (Poison Training Data)
Defender implication: Data provenance and integrity controls matter as much as model accuracy. Know where your training data comes from, who can modify it, and whether any data sources are accessible to external parties.
Privacy Attacks
Privacy attacks extract information that should not be accessible from a trained model. The two main categories are:
Model inversion: An attacker queries the model with carefully crafted inputs to reconstruct information about the training data. In healthcare or financial contexts, this can mean recovering sensitive personal information from a model trained on private records.
Membership inference: An attacker determines whether a specific data point was included in the model's training set. This can violate data protection requirements and confirm sensitive information about individuals.
Model extraction: An attacker queries the model extensively to build a local copy (a shadow model) that approximates the original's behaviour. The shadow model can then be used for offline adversarial attack development without triggering rate limiting or monitoring on the original.
MITRE ATLAS technique: AML.T0024 (Infer Training Data Membership), AML.T0025 (Model Inversion Attack)
Defender implication: Rate limiting, query monitoring, and differential privacy techniques all apply. A model that can be queried without restriction or monitoring is significantly more vulnerable to all three privacy attack types.
Abuse Attacks
Abuse attacks use ML systems as intended - but for purposes their operators did not authorise. Prompt injection is the most operationally significant abuse attack for defenders in 2026: manipulating an LLM's behaviour through crafted input that overrides system instructions or causes the model to take unintended actions.
The LiteLLM incident in March 2026 illustrated how supply chain compromise affecting ML infrastructure can produce a large blast radius across all dependent deployments. Supply chain compromise targeting ML dependencies, training frameworks, and inference servers is a documented and growing attack vector.
MITRE ATLAS technique: AML.T0051 (LLM Prompt Injection), AML.T0010 (ML Supply Chain Compromise)
Defender implication: Input validation, output filtering, privilege separation, and supply chain security controls all apply to ML systems. Treat LLM applications as you would any other privileged software component.
The Adversarial ML Attack Reference Table
| Attack category | When it occurs | What it targets | SOC/defender impact | MITRE ATLAS | Primary mitigation |
|---|---|---|---|---|---|
| Evasion | Inference time | Model decision boundary | ML-based malware detectors, network anomaly detection systems evaded | AML.T0015 | Adversarial robustness testing, ensemble detection, human-in-the-loop validation |
| Poisoning | Training time | Training data and learned model behaviour | Threat intelligence models learn to misclassify attacker-controlled samples as benign | AML.T0020 | Data provenance controls, input validation on training data, anomaly detection on training sets |
| Model inversion | Inference time | Training data privacy | Sensitive data recoverable from models trained on private records | AML.T0025 | Differential privacy, query rate limiting, output confidence score suppression |
| Model extraction | Inference time | Model intellectual property and decision logic | Shadow models used for offline adversarial attack development | AML.T0016 | Query monitoring, rate limiting, watermarking model outputs |
| Prompt injection | Inference time | LLM instruction following | AI agents take unintended actions; safety controls bypassed; data exfiltrated | AML.T0051 | Input validation, privilege separation, output filtering, least-privilege tool access for agents |
| Supply chain compromise | Pre-training / deployment | ML dependencies and infrastructure | Compromised ML libraries affect all dependent deployments simultaneously | AML.T0010 | Software composition analysis, dependency integrity verification, internal approved library registry |
How Do You Learn Adversarial Machine Learning as a Cyber Defender?
The challenge with most adversarial ML learning resources is that they are written for ML researchers, not security practitioners. They assume familiarity with neural network architecture, gradient descent, and loss functions before getting to the attack techniques that matter operationally.
TryHackMe's AI Security path is built for security practitioners. It covers adversarial ML concepts through the defender's lens: not how to train a model from scratch, but how to understand where models break, how attackers exploit those breaks, and what controls apply. The 25 hands-on labs put you inside live AI systems working with the techniques that matter operationally.
Specific rooms directly relevant to adversarial ML defenders:
AI Threat Modelling covers MITRE ATLAS-based assessment: mapping your AI systems' attack surface against the ATLAS tactics and techniques matrix, prioritising the techniques most relevant to your deployment, and designing controls against them.
LLM Security covers prompt injection and jailbreaking from both the attacker and defender perspective. You run the attacks against live LLM systems and then apply the defensive controls that reduce exposure.
AI Supply Chain Security covers dependency integrity, supply chain attack vectors, and how to assess and secure the ML libraries and infrastructure your deployments depend on.
AI Forensics covers what investigation looks like when an AI system has been attacked: what artefacts to collect, how to determine whether a model was poisoned, and how to reconstruct what happened.
AI1, TryHackMe's AI Security certification, validates this skill set across 13 hands-on scenarios covering both offensive and defensive AI security. It is the first practical AI security credential available on any platform. Premium subscribers receive a 15% discount.
FAQ
How is AI being used in cyber security attacks and defences? On the offensive side: AI-generated phishing that is indistinguishable from legitimate correspondence, adaptive malware that modifies its own code to evade signature-based detection, automated vulnerability discovery at a scale and speed that human researchers cannot match, and deepfake-based fraud that bypasses traditional social engineering awareness. On the defensive side: ML-based anomaly detection that identifies threats without requiring known signatures, AI-assisted alert triage that reduces the volume analysts need to review manually, automated threat intelligence enrichment, and LLM-powered investigation assistance that accelerates root cause analysis. Both sides are using the same underlying technology. The defenders who understand the offensive applications build better detection logic.
How do you defend against AI-powered cyber attacks? Defending against AI-powered attacks requires controls at multiple layers. For AI-generated social engineering: procedural verification through a separate channel for any high-value request, regardless of how legitimate the initiating message appears. For adaptive malware evasion: behaviour-based detection that identifies what a process is doing rather than matching against known signatures, combined with heuristic analysis that catches novel variants. For adversarial ML attacks against your own AI systems: adversarial robustness testing as part of your model validation pipeline, data provenance controls for training data, and monitoring for anomalous query patterns that suggest model extraction or evasion attempts. For AI agents with tool access: least privilege on tool permissions, input validation, and output filtering before any agent action takes external effect.
What is MITRE ATLAS and how does it relate to MITRE ATT&CK? MITRE ATLAS is the AI-specific counterpart to MITRE ATT&CK. ATT&CK maps adversary tactics and techniques against traditional IT infrastructure. ATLAS maps them against AI and ML systems. As of February 2026, ATLAS v5.4.0 contains 16 tactics, 84 techniques, 56 sub-techniques, and 42 real-world case studies. If you already use ATT&CK for threat modelling and detection engineering, ATLAS applies the same structured approach to your AI attack surface. Approximately 70% of ATLAS mitigations map to existing security controls, making integration with current SOC workflows practical rather than requiring entirely new defensive infrastructure.
Nick O'Grady