
A safe agent design defines not only what the agent can do, but also when it must stop. This is especially important in security workflows, where an investigation-support agent may gather evidence, compare signals, and recommend a priority, but should stop when evidence is missing, contradictory, or outside its scope. At NorthStar Fashion, the engineer remains accountable for the final decision: the agent supports the investigation without replacing human judgement. Human oversight and clear stop conditions therefore keep the agent within its intended boundary.
The Autonomy Problem
Agents are useful because they can make decisions during execution - such as selecting a tool, inspecting its result, updating state, and determining the next step - but that same flexibility also creates risk.
If an agent is not given clear limits, it may:
- Keep retrying when evidence is unavailable.
- Treat missing evidence as safe.
- Ignore contradictory results.
- Produce a confident answer without support.
- Recommend actions outside its authority.
- Continue working outside the original scope
A well-designed agent should not attempt to complete a task at any cost; when evidence is insufficient, risk is too high, or the situation requires human judgement, the safest action is to stop and request human review.
What Human Oversight Means

Human oversight means that a person remains responsible for reviewing the agent’s output and making the final decision. This does not make the agent less useful: it can still reduce manual effort by collecting evidence, organising context, identifying missing information, and preparing a structured summary. The key distinction is that the agent supports the decision-making process, while the engineer retains ownership of the decision.
For NorthStar Fashion, the agent may say:
Related alerts and reputation data suggest this alert may require higher priority. Human review is required before any response action.
But it must not say:
This is malicious. The account has been disabled.
The first response supports the engineer’s decision-making, while the second performs - or implies - the kind of action that falls outside the agent’s intended scope.
Human Review Points
A human review point is a condition that requires the engineer’s involvement before the workflow can continue or the agent’s output can be accepted. For this design, human review should be required when:
- Evidence is missing.
- Evidence is contradictory.
- The alert may indicate account compromise.
- The recommended priority is high or critical.
- The agent has low confidence.
- A required source is unavailable.
- The agent cannot explain its recommendation
These conditions matter because unsupported or automated conclusions could create security or business risk. For example, if an IP reputation source is unavailable, the agent should not assume the IP is safe; it should record the evidence as missing and require human review. Similarly, if related alerts indicate a possible account compromise, the agent should escalate the finding to the engineer rather than disabling the account itself.
Stop Conditions
A stop condition tells the agent when it must stop the workflow instead of continuing automatically. Stop conditions are different from human review points, but they often work together.
A human review point says:
A person must review this.
A stop condition says:
The agent must not continue past this point.
For NorthStar Fashion, the agent should stop when:
- Required evidence cannot be retrieved.
- Repeated tool errors occur.
- Evidence is contradictory.
- The request is outside scope.
- The maximum retry limit is reached.
- Human review is required.
- The agent lacks enough evidence to support a recommendation
This prevents the agent from guessing, looping indefinitely, or producing a recommendation that appears more certain than the evidence allows.
Retry Limits
Retries can help recover from temporary tool failures, such as a timeout during an IP reputation lookup, but they must have clear limits. Without a retry limit, the agent may waste time, increase costs, fill logs with repeated failures, and delay the engineer from reviewing the alert manually.
A safer pattern is:
Try the approved tool. If the tool fails, retry a limited number of times. If the tool still fails, mark the evidence as unavailable. Stop or request human review.
The agent should never retry forever.
Uncertainty Is a Valid Outcome
An unsafe design assumes that the agent must always produce a final answer, whereas a safer design allows it to acknowledge when the available evidence is incomplete.
For example:
The IP reputation source could not be reached. Related alerts were found, but the reputation evidence is missing. Human review is required before assigning final priority.
This is safer than inventing a conclusion or treating an unavailable source as evidence that no risk exists. In security workflows, uncertainty should remain visible, so the final output should clearly identify missing evidence, known limitations, the confidence level, and any human-review requirements.
Safe vs Unsafe Behaviour
The agent should be designed to fail safely. A safe failure is one where the agent stops, explains what happened, and asks for review instead of continuing with unsupported assumptions.
Design Principle: Safe Stop
Human oversight and stop conditions should follow this principle:
The agent may support investigation, but it must stop when the evidence no longer supports safe continuation.
This principle keeps the agent’s autonomy within clear boundaries: it may collect and organise information, but it must not continue blindly, conceal uncertainty, or make final security decisions.
