Why AI Pilots Stall in the SOC, Even When the Demo Works

The AI demo looks excellent. It processes an alert, builds a timeline, summarises the evidence and recommends what the analyst should investigate next.

Everyone agrees it could save time. Six weeks later, the SOC is still running a pilot.

The problem might be an identity integration that hasn't been approved, a reviewer who doesn't have the capacity to check the output, or a disagreement about what the AI should be allowed to do.

Whatever the cause, the team has found itself in an uncomfortable position. The technology appears capable, but putting it into everyday operations remains difficult.

This gap comes through in our research, which examines how security organisations are adopting AI and where they continue to draw the line on its authority.

Across 43 organisations analysed in depth, we identified 78 distinct AI use cases. Of those, 21 were in production, 19 were being piloted and 38 were still exploratory.

For SOC leaders, the question is how to move from a promising demonstration to a reliable operating capability.

The Demo Is the Easy Part

A controlled demonstration can show what an AI system does when the data is available, the integration works and somebody has already prepared the task.

The SOC has to deal with everything that happens outside those conditions.

Consider an AI tool designed to investigate suspicious access to a customer-facing application. During the pilot, it receives the relevant logs, identifies unusual behaviour and produces a convincing summary.

But when the tool is connected to the organisation's normal environment, the investigation depends on identity records stored elsewhere. The connector lacks permission to retrieve them, and the team responsible for that system hasn't approved the request.

The model can still produce a summary. It just cannot establish everything the analyst needs to know.

This is one of the central challenges of using AI in everyday SOC operations. A useful answer depends on the quality and availability of the information behind it.

Before expanding a pilot, identify the data sources required for the task and check whether the production integration can retrieve them consistently.

That means testing the details that rarely feature in a demonstration: incomplete API responses, expired credentials, missing records, inconsistent asset identifiers and service interruptions.

An integration that works during a supervised trial may still need considerable engineering before a SOC can rely on it throughout the week.

The Barriers Are Often Outside the Model

Our research helps explain why AI projects can lose momentum after an encouraging start.

Across the 43 organisations, several recurring obstacles appeared in discussions about AI in security operations.

The findings point towards a broader operational challenge.

Teams need people capable of evaluating AI output, enough capacity to introduce and maintain the workflow, reliable access to data and agreement on who is accountable when something goes wrong.

Those dependencies often involve different departments and decision-makers.

A SOC manager might support deployment while the identity team is still reviewing permissions. Security engineering may have completed the integration while risk owners remain uncomfortable with the actions the system can initiate.

The pilot continues because the organisation has yet to resolve the conditions required for production.

Another month of model testing may produce better answers without resolving any of those dependencies.

Who Actually Has Permission to Make the Decision?

One of the quickest ways to complicate an AI pilot is to leave its authority undefined.

Consider three possible uses of AI during an investigation.

An AI system might draft an incident summary for an analyst to review. It might recommend disabling an account after identifying suspicious behaviour. Or it might be permitted to disable that account itself.

Each involves different consequences and therefore different approval requirements.

In our research, none of the 21 production AI use cases described a model independently deciding high-consequence containment, eradication or recovery actions.

Some organisations did describe automated containment through predefined playbooks. As we've explored in our comparison of SOAR playbooks and agentic AI, an action triggered by human-defined rules is different from an AI model independently deciding that the action is justified.

For a SOC leader, the approval boundary should be part of the pilot specification.

Establish which systems the AI can access, what information it can retrieve, which actions require human approval and who can suspend the workflow.

An organisation may be comfortable allowing AI to prepare an escalation while remaining unwilling to authorise changes to production systems.

AI Can Save Time and Still Create More Work

A pilot may reduce the time required to write an initial investigation summary while adding work elsewhere.

Someone has to check the answer.

If the AI misses relevant evidence, attributes an event to the wrong system or reaches a conclusion the logs cannot support, an analyst needs to find and correct the problem.

That review may be relatively quick for straightforward cases. More complicated investigations can require substantial effort, particularly when the AI's explanation is persuasive.

Our experiment with an agentic SOC working through tabletop investigations illustrates the difficulty. The agents could produce confident, detailed reasoning even when they lacked sufficient evidence to support a decision.

Fluent analysis is easy to mistake for reliable judgement.

This creates a practical requirement for SOC teams: analysts need to understand the underlying investigation well enough to challenge the AI's conclusions.

But they also need the time to do it.

A pilot evaluated by a senior analyst working closely with the implementation team may perform very differently when handed to an already busy duty shift.

Measure the entire investigation workflow, including preparation, verification, corrections, escalation and any additional support required.

If AI reduces the initial drafting task from ten minutes to three but introduces eight minutes of review and correction, the operational benefit looks different from the demonstration.

Include investigation quality alongside the time measurements. A faster case closure is of limited value if the next responder receives an incomplete or misleading handover.

Stop Adding Features Before the First One Works

There's another common reason pilots drift. The original use case keeps expanding.

A tool that begins by summarising cloud alerts is soon asked to retrieve identity information, investigate unusual API activity, recommend containment and produce management reports.

Each addition sounds reasonable. Together, they turn a manageable evaluation into a much larger implementation.

More data sources require more permissions and engineering. More consequential recommendations introduce additional testing and governance requirements.

A SOC can find itself repeatedly changing what success means before it has established whether the original workflow works.

Keep the first deployment decision narrow. For example, the team might agree to evaluate AI-generated summaries for one defined alert queue, with analyst review required before case closure.

Proposed additions can be documented for later assessment.

If a change is essential, update the scope and repeat the relevant tests. Otherwise, let the current evaluation reach a conclusion.

Five Questions That Should Decide Whether Your Pilot Goes Live

A pilot review should be a decision meeting with evidence on the table.

Before approving production use, ask:

Set the acceptance criteria according to the specific use case and its consequences.

A tool that prepares draft reports can operate under different conditions from one that recommends disabling privileged accounts.

The review should lead to one of four decisions: deploy within the approved scope, narrow the scope, extend the pilot to resolve a specific issue or stop the evaluation.

If the team needs more time, identify exactly what remains unresolved and who owns it. Otherwise, the pilot can drift into another cycle of demonstrations without bringing the SOC closer to a decision.

Stopping a pilot may also be the right choice when the review burden exceeds the benefit.

Production Starts When the Pilot Team Leaves

A capable model and a successful technical integration are important milestones. Keeping the workflow reliable after launch requires ongoing ownership.

The SOC needs to know who maintains the connectors, handles failures, reviews model changes and decides when the service should be suspended.

Analysts need a clear fallback when the AI is unavailable or its output cannot be trusted.

Changes to the data sources, prompts, model or permissions should be assessed for their effect on the approved workflow.

This is particularly important because an AI system can behave differently when the information supplied to it changes, even if the model itself remains the same.

The organisation should also retain enough evidence to investigate mistakes and determine whether corrective changes have worked.

Production readiness is therefore partly a question of whether the SOC can operate the capability without depending on the people who built the pilot.

The Hard Part Begins After the Demo

AI can make security investigations faster and help analysts work through information that would otherwise consume valuable time.

But the move into production exposes questions a demonstration can leave unanswered.

Can the system access the evidence it needs? Do analysts have the capacity and judgement to verify its conclusions? Has the organisation agreed where automated assistance ends and human authority begins?

Those questions sit at the heart of our research into how security teams are adopting AI and where they continue to draw boundaries around consequential decisions.

The report examines 78 AI use cases across 43 organisations, including what has reached production, where pilots remain exploratory and which operational barriers teams describe.

For SOC leaders deciding where to invest next, the findings offer a useful perspective on the work that must happen around the technology.

And for teams developing that capability, realistic SOC investigations provide opportunities to practise evidence analysis, escalation and decision-making before those skills are tested during a live incident.