AI Security Threats

Welcome back. Room 1 covered the technology stack that powers modern AI: how ML algorithms learn from data, how neural networks process it, and how LLMs like ChatGPT emerged from all of that. If any of those terms are still fuzzy, it's worth going back before continuing.

This room is where things get interesting from a security perspective. Now that you understand what AI is and how it works, we're going to look at what happens when it goes wrong, intentionally or otherwise. This room covers:

• The vulnerabilities that AI models introduce into an organisation's attack surface.
• How attackers are enhancing existing techniques using AI.
• How defenders are fighting back with the same technology.
• What it means to adopt AI securely.

The rate at which AI has exploded onto the scene has left a lot of security teams playing catch-up. By the end of this room, you'll understand the threat landscape well enough to stop doing that.

Learning Prerequisites

This room requires completion of Room 1: The Building Blocks of AI, or equivalent knowledge of AI, ML, neural networks, and LLMs.

Learning Objectives

• Understand the key vulnerabilities that AI models introduce and how attackers exploit them.
• Understand how AI is being used to enhance existing attacks like phishing, malware generation, and social engineering.
• Understand how AI can be used defensively across analysis, prediction, summarisation, and investigation.
• Understand what it means to adopt AI securely and the frameworks that guide that process.

Learn the New Threats

Now that AI is embedded in business operations across every industry, it's introduced a new category of security concern: vulnerabilities that are specific to AI models themselves. These aren't the same as traditional software vulnerabilities. They emerge from the nature of how these models are built, trained, and deployed. To help us make sense of them, we can lean on a familiar friend.

If you've spent any time in cyber security, you've probably come across the MITRE ATT&CK framework. MITRE have built something similar with a focus specifically on AI threats, called the ATLAS framework. It maps out the tactics, techniques, and procedures attackers use against AI systems, and it's a useful reference as you work through this room. You can check it out here (opens in new tab).

Vulnerability Breakdown

Let's look at the five key vulnerabilities in AI models that every security practitioner should know.

Prompt Injection occurs when an attacker overrides the original instructions provided to a model. Every AI model operates under a system prompt, a set of instructions that define how it should behave. An RPG chatbot might be told to stay in character and never discuss its underlying infrastructure. Prompt injection is when user input is crafted in a way that overrides or bypasses those instructions, causing the model to behave in ways it wasn't supposed to, whether that's revealing sensitive information, generating harmful content, or acting outside its defined scope.

Data Poisoning is when an attacker manipulates the training data used to build an AI model, causing its outputs to be incorrect or biased. Take a spam filter trained on email data. If an attacker can tamper with that training data before the model is trained, they can cause the model to misclassify spam as legitimate mail, effectively blinding it to the very emails it was built to catch.

Model Theft occurs when an attacker gains unauthorised access to an AI model, either to steal the intellectual property it represents or to use it for malicious purposes. One method is to repeatedly query a model's API and use the outputs to train a clone that replicates its behaviour, without ever needing direct access to the original weights.

Privacy Leakage refers to the possibility of an AI model inadvertently revealing sensitive information from its training data. A model trained on private medical records, for example, could under the right prompting conditions surface details about real patients that were never intended to be accessible. The information doesn't disappear when training ends; it gets baked into the model's weights.

Model Drift is when a model's performance degrades over time as the world it was trained on changes. A model trained on last year's network traffic patterns may start performing poorly as attack techniques evolve. This is why monitoring deployed models isn't optional; it's a security requirement. Drift can go undetected until the model is already failing in production.

Now it's your turn to get hands-on with these vulnerabilities. Click the Open Agent button above to access a deliberately vulnerable AI assistant deployed inside a fictional company. You've been given five objectives, one per vulnerability. Work through each one, successfully exploit the weakness, and collect a fragment of the flag for each success.

The vulnerabilities covered in the previous task are threats that emerge from AI being present in a system. This task covers a different angle: existing attacks that have been made significantly more dangerous by giving attackers access to AI tools. These aren't new attack categories. They're familiar techniques with a serious upgrade.

AI-Generated Malware

Generative AI can produce functional code in seconds from a natural language prompt. That's an enormous productivity boost for developers, and it's an equally enormous productivity boost for attackers. Writing malware has historically required technical skill and time. With generative AI, that barrier drops considerably. Attackers can generate, iterate, and customise malicious code faster than ever, and the models doing the generating have no way to verify the intent behind the request.

Deepfakes

Authentication, at its core, is about answering one question: are you who you say you are? For most of human history, seeing and hearing someone was enough to answer it. Generative AI has broken that assumption. Given enough training data, an AI can now generate a convincing likeness of a real person, whether that's their voice, their face, or both, to a degree of accuracy that fools even technically aware individuals.

The attack scenario practically writes itself. A finance employee receives a voice message from what sounds exactly like their CEO, requesting an urgent wire transfer. The voice is a deepfake. Examples of this already being used in the wild include deepfaked video interviews that led to fraudulent job offers being extended to candidates who didn't exist. The technology is advancing faster than our ability to detect it.

AI-Enhanced Phishing

Phishing is one of the most common initial access methods in use today. For years, security awareness training gave defenders a fighting chance by teaching people to spot the telltale signs: suspicious links, urgency, and, perhaps most reliably, broken or unnatural language. That last indicator is becoming obsolete. Generative AI can produce fluent, contextually appropriate, highly targeted phishing emails at scale and with minimal effort, regardless of the attacker's own writing ability.

Most LLMs have guardrails designed to prevent them from generating obviously malicious content. But as covered in the previous task, prompt injection techniques can sometimes be used to bypass those guardrails, making the same models that power productivity tools available to attackers as phishing content generators.

Time to work the other side of the desk. Click the Open Agent button above to open your SOC inbox. A queue of messages is waiting for you, including AI-generated phishing emails, a deepfake voice transcript, and an AI-assisted social engineering attempt. Triage each one, identify the threat, and flag the AI enhancement being used. Clear the full inbox to get your flag.

Harness the Power

It would be easy to read the last two tasks and come away feeling like AI is purely a threat. It isn't. The same technology that gives attackers new capabilities gives defenders something far more valuable: scale. Let's look at the numbers first.

IBM's annual Cost of a Data Breach report (opens in new tab) found that organisations that had adopted AI saved an average of $2.2 million per breach compared to those that hadn't. Given that the average breach cost in the same report sat at $4.88 million, that's not a marginal gain. The same report found that AI-assisted teams identified and contained breaches 108 days faster than those without it. The conclusion is pretty clear: adopting AI isn't just a nice-to-have, it's a competitive security advantage.

Here are four areas where AI has a direct and measurable impact on defensive security operations.

Analysis: A huge proportion of security work is pattern recognition at scale: finding anomalies in network traffic, spotting unusual authentication behaviour, identifying suspicious process activity in logs. This is exactly what ML was built for. Products like Microsoft Defender for Endpoint and Splunk already leverage AI to analyse input data and surface anomalies at speeds no human analyst could match. The 108-day improvement in breach detection time starts to make sense when you consider what AI can do to the analysis problem.

Prediction: AI models trained on historical attack data can begin to predict future threats before they fully materialise. Consider phishing, one of the attack types covered in the previous task. The same AI capabilities that make phishing emails harder to spot can be turned around and used to detect them. A model trained on vast volumes of phishing examples can identify patterns in email content that a human reviewer would miss, and once it's made a prediction, it can automate the response, blocking the email before it ever reaches a user's inbox.

Summarisation: Security incidents generate a huge volume of artefacts: logs, reports, alerts, threat intelligence. Reading and synthesising all of that takes time that defenders often don't have. LLMs can summarise incident reports, extract the key findings from lengthy documents, and draw correlations between events that a human analyst under pressure might miss entirely. That time saving compounds quickly across a busy SOC.

Investigation: When something goes wrong, working out what happened and why is a core security function. LLMs can be fed raw logs and asked to explain what they show, suggest queries to run, and help triage an active incident in natural language. They're also useful for threat hunting, which relies heavily on imagination: thinking up attack scenarios an adversary might use that defenders haven't considered yet. AI can surface possibilities that simply wouldn't have occurred to a human analyst working alone.

Now you get to use these capabilities for real. Click the Open Agent button above to launch Incident Zero, a live simulated security incident unfolding in real time. Your AI co-pilot is standing by. Work through the incident using all four defensive capabilities covered above: analyse a log, triage a phishing report, get a summary of the incident brief, and pull threat hunting leads. Complete the full response chain to get your flag.

The New Frontier

AI adoption in cyber security is the right call. The data from the IBM (opens in new tab)report makes that clear. But there's a catch that the same report also flags: only 24% of generative AI initiatives are currently secured. Adopting AI without securing it doesn't just fail to reduce risk. It actively introduces new attack surface. The vulnerabilities covered in Task 2 don't disappear because you're using AI defensively. They show up whether you're the attacker or the defender, and they need to be addressed from the moment AI enters your environment.

Here's what good AI security hygiene looks like in practice.

Securing AI Models: Many of the model vulnerabilities discussed earlier, prompt injection, privacy leakage, model theft, share a common thread: they involve an attacker getting access to something they shouldn't. The first line of defence is controlling who can interact with your AI systems in the first place. Implementing strong authentication, defining strict access permissions, and using RBAC (Role-Based Access Control) and MFA (Multi-Factor Authentication) significantly reduces the attack surface at the model interaction layer.

Privacy Protection: Training data frequently contains sensitive information, whether that was intentional or not. Patient records, internal communications, and customer data can all end up baked into model weights if the training pipeline isn't properly governed. Training data should be treated with the same care as any other sensitive data asset: audited, minimised, and encrypted.

AI Security Standards: Frameworks exist specifically to guide the secure development, deployment, and maintenance of AI systems. ISO/IEC 27090, for example, provides guidance on identifying and mitigating security threats specific to AI. Incorporating established standards throughout the AI lifecycle means organisations can get ahead of risks rather than discovering them in production.

Model Monitoring: Monitoring a deployed model isn't just about catching performance degradation or flagging when retraining is needed. It's a security function. Unexpected behaviour, anomalous outputs, and statistical drift can all be indicators of an active attack. Explainability tools like SHAP and LIME help make model behaviour more interpretable, giving security teams visibility into what the model is actually doing rather than treating it as a black box.

The message is simple: adopt AI fast, because the window where attackers have it and defenders don't is a dangerous place to be. But adopt it with the same rigour you'd apply to any other system in your environment. The benefits are real and significant. So are the risks if you skip the security fundamentals.

Graduation Time

You've covered the full AI security landscape across both rooms. Now it's time to prove it.

Click the View Site button above to launch the AI Security Analyst Orientation.

A pixelated AI bouncer is standing between you and your AI Fundamentals Licence, and he's not letting anyone through without credentials. Work through the exam, pass with the required score, and your licence, complete with flag, will be waiting on the other side.

Everything you need to answer the questions is in this room and Room 1. Good luck.

Across both rooms you've built up a complete picture of where AI has come from, how it works, and what it means for security on both sides of the fence. Here's a recap of what's been covered:

• Artificial Intelligence is the overarching field concerned with enabling machines to simulate human intelligence, with roots going back to the 1950s.
• Machine Learning is a subfield of AI in which models learn from data through a structured lifecycle, using algorithms that fall into four categories: supervised, unsupervised, semi-supervised, and reinforcement learning.
• Neural networks replicate the structure of the human brain through layers of weighted nodes, enabling increasingly complex feature extraction. Networks with more than three layers qualify as Deep Learning.
• Large Language Models are built on transformer neural networks, trained on vast datasets through pre-training and refined through RLHF, predicting the next word in a sequence to generate human-like text.
• AI introduces a new category of model-specific vulnerabilities including prompt injection, data poisoning, model theft, privacy leakage, and model drift, mapped by the MITRE ATLAS framework.
• Attackers are using AI to enhance existing techniques, making malware generation, deepfakes, and phishing faster, cheaper, and harder to detect.
• Defenders can use AI to enhance analysis, prediction, summarisation, and investigation, with IBM data (opens in new tab) showing AI-assisted teams contain breaches 108 days faster and save an average of $2.2 million per incident.
• AI adoption needs to be done securely from day one, with proper access controls, training data governance, adherence to standards like ISO/IEC 27090, and ongoing model monitoring.

From here, the path goes deeper. The next rooms in the AI Security learning path build on everything covered here, getting into the specifics of how these attacks work, how to test for them, and how to defend against them properly.