The vendor closes an alert in 90 seconds. The dashboard flashes green, the case summary reads beautifully, and everyone moves on to the next slide.
Nobody asks the expensive question: who decided it was safe to close?
Perhaps the AI investigated the alert and reached its own conclusion. Perhaps a predefined playbook handled the case, or an analyst approved the decision behind the scenes. All three can produce the same impressive demonstration, but they give the customer very different levels of automation, responsibility and risk.
That's where buying an autonomous security operations centre (SOC) becomes complicated.
Vendors increasingly promise automated triage, investigations and incident response. Some products can perform substantial investigative work, while others combine AI-generated analysis with conventional security orchestration. Their capabilities may overlap, but the authority granted to each system is what determines how far it can act without human intervention.
For CISOs and SOC directors, the buying process needs to establish what the technology can reliably accomplish, which decisions it controls and how much work the security team will still need to perform.
Here is how to get past the polished demonstration and evaluate the system you would actually deploy.
A Closed Alert Doesn't Tell the Whole Story
Imagine a product claiming to resolve 80% of security alerts automatically. It sounds like a substantial reduction in analyst workload. Before accepting the figure, ask what counted towards that 80%.
Did the system investigate each alert and establish whether the activity was malicious? Did it suppress duplicates, close known false positives or route routine cases through existing rules?
How many alerts were excluded because the system lacked sufficient information or couldn't handle the investigation?
The answers determine what the number means.
A useful evaluation starts by establishing the exact work the vendor claims to automate. Define the initial alert, the steps the system performs and the condition that counts as a completed investigation.
For a SOC buyer, that means asking the vendor to demonstrate the proposed workflow against the information, constraints and decisions your own team encounters.
Follow the Decision
Autonomy becomes much easier to evaluate when you trace how the system reaches an action. Suppose an AI-enabled SOC product investigates suspicious activity on a development server and recommends isolating it from the network.
That decision could follow several paths.
- A conventional SOAR playbook might isolate the server because predefined conditions were met.
- An AI agent might reach the same recommendation after analysing the evidence, then send it to an analyst for approval.
- A more autonomous system might select and execute the action within its assigned permissions.
While the resulting action could look identical, the difference lies in who made the decision and which controls governed it.
As we've explored in our comparison of SOAR playbooks and agentic AI, predefined automation and model-driven decisions can operate together. An AI classification can trigger a traditional playbook, meaning the model's judgement still influences the outcome even when the final action follows a fixed rule.
During procurement, ask the vendor to trace one consequential action from the original evidence to execution.
You should be able to identify what information the system used, which component selected the action, where authorisation occurred and what happened afterwards.
Where SOC Teams Are Drawing the Line
This question also appears in TryHackMe's original research, The Hard Part Is Still Human, which examines how organisations are using AI in security operations.
Across 43 organisations analysed in depth, we identified 78 distinct AI use cases, including 21 in production.
Among those 21 production use cases, none described an AI model independently deciding high-consequence containment, eradication or recovery actions.
Explore what our research reveals about trust, verification and human judgement in the SOC.
Give the AI an Investigation That Refuses to Cooperate
Most product demonstrations are carefully prepared. The relevant data is available, the integrations work and the system follows a scenario it can complete.
Real investigations are rarely so accommodating. A proof of value should therefore include cases selected by the customer, preferably drawn from historical incidents or authorised test environments.
Keep some scenarios outside the examples the vendor uses repeatedly for tuning. Establish the expected outcomes before the evaluation begins and record material configuration changes during testing.
Then introduce the conditions that make a real investigation difficult.
Start with an ambiguous case
A development server generates unusual network activity shortly after a deployment. The activity resembles a possible compromise, but it may also be part of an authorised release process. Can the system retrieve the necessary context and reach a defensible conclusion?
Introduce conflicting evidence
Endpoint activity suggests one explanation, while identity and application records tell a different story. Does the AI investigate the inconsistency, or produce a confident summary from whichever source it encountered first?
Remove a dependency
Make a required data source unavailable or return an incomplete response. The system should recognise that its investigation has been limited and communicate what remains unknown.
These tests reveal how the product behaves when the evidence is incomplete and the correct answer requires judgement.
They also expose a weakness that polished AI explanations can conceal.
In a TryHackMe experiment involving five AI agents working through SOC tabletop investigations, the agents frequently produced confident reasoning during early, ambiguous stages despite making weaker decisions.
Explore what our research reveals about trust, verification and human judgement in the SOC.
Test What Happens When the Agent Oversteps
Any system authorised to interact with production security tools needs a reliable way to reject inappropriate actions.
For example, an autonomous SOC product might be permitted to isolate ordinary employee workstations while critical infrastructure remains outside its approved containment scope.
During the evaluation, introduce a scenario in which the AI recommends isolating an excluded asset. The product should demonstrate that its execution controls prevent the action, even if the model argues that containment is necessary.
Six Questions Worth Bringing Into the Buying Meeting
A procurement scorecard should make claims easier to verify and differences between vendors easier to explain.
Also document the support each vendor provides during the trial. An engineer fixing integrations in the background may be reasonable during implementation, but buyers need to know what support continues after purchase.
The Best Demo Is One You Can Challenge
Across the organisations we studied, AI adoption raised questions about trust, operational capacity, integration and the authority to make consequential decisions.
The report examines those barriers, the AI use cases reaching production and the boundaries security teams are establishing around autonomous response.
For a SOC leader evaluating vendors, those findings provide useful context for deciding how much authority to delegate and what the team must still be prepared to do.
That human capability deserves attention alongside the procurement process. Realistic incident response exercises give teams opportunities to investigate evidence, coordinate decisions and practise containment under controlled conditions.
Explore what our research reveals about trust, verification and human judgement in the SOC.
Ayomide Joseph A.
