AI agents: how much autonomy should you give them?

For a long time, using artificial intelligence in a company meant asking it a question and waiting for an answer.

With agents, the relationship changes. You no longer just ask how something is done: you give access to tools and ask for it to be done. Find a piece of information, edit a file, query an application, draft a reply, trigger a workflow, chain several operations together until an objective is met.

The difference may look technical. It is mostly operational.

A wrong answer and a wrong action do not have the same consequences.

Having spent two years demonstrating what agents can do, companies now have to decide what they are willing to let them do alone. At what level of reliability, and at what cost.

An agent is not just a more sophisticated chatbot

A chatbot answers. An agent pursues an objective.

In practice it breaks the task down, picks a tool, runs a search, interprets the result, decides on the next step — and starts again until it judges the job done.

In conventional software, the execution path is written in advance. With an agent, part of that path depends on decisions taken during execution.

So the company no longer controls code alone. It also has to control a behaviour.

That is a change in kind, not in degree. The testing practices you apply to deterministic software stop being enough when two identical runs can take two different routes.

A successful demo does not prove reliability

A demo answers a simple question: can the agent do this task?

Production asks a different one: can it do it correctly, often enough, including in situations we have not all anticipated?

These are not the same thing. Capability is shown once; reliability is established over time. It covers the success rate, but also consistency from one run to the next, resistance to unusual inputs, the ability to recognise that it does not know — and the severity of errors when they happen, which is not the same whether you get the tone wrong or the bank account.

Princeton's work on agent evaluation documented that gap: agents that shine on a benchmark turn out to be expensive, hard to reproduce or brittle as soon as you step outside the protocol. The problem is not the headline performance. It is what that figure leaves out.

Add a mechanism that most people underestimate: over a long task, errors compound. An agent that is fine at each step taken separately becomes markedly less reliable when it has to chain twenty of them. The success rate of a sequence is not the success rate of its links.

So testing only the final output is no longer enough.

Quality also lives between the steps

Take an agent handling an internal support request.

It has to understand the request, identify the employee, find the applicable rules, sometimes consult several systems, then propose or carry out an action.

The result can be right while the path was wrong. The agent consulted the wrong source and happened to land correctly. It used stale data that happened to still be valid. It called four tools where one would have done. It skipped a rule nobody checked, because the answer looked fine.

None of this shows up when you compare an input with an output. It shows up when you look at what happened in between.

That is precisely the subject of Anthropic's work on evaluating agents: assessing the trajectory as much as the result — the tools called, the order of operations, the actions actually performed. An agent that reaches the right result by the wrong route will eventually reach the wrong result by that same route.

Quality assurance therefore has to move. It no longer checks an input-output pair: it defines what counts as acceptable behaviour during execution, and what does not.

Autonomy is not a switch

The question is rarely "supervised or autonomous". Still less across a whole system.

The same agent can look something up on its own without any difficulty, prepare a decision a human will approve, automatically carry out a reversible operation, and stop dead in front of a financial or irreversible action. Four regimes, one agent.

What sets the regime comes down to a few questions. What does an error cost? Can you undo it, and at what price? How much confidence do you have — measured, not assumed — on this particular kind of case? And what does the business say: a recoverable slip inside a back office is not one in front of a customer or a regulator.

The World Economic Forum argues for governance calibrated to the level of autonomy actually granted, rather than a single regime applied to an entire system. Gartner, for its part, observes that today's difficulty lies less in what agentic AI promises than in telling apart what it already delivers from what remains emerging.

Autonomy should not be granted once and for all. It should be earned.

Observe first. Then recommend. Then act on approval. Then, on a narrow and instrumented scope, act alone. Each step up is justified by results, not by a decision of principle taken in a steering committee.

Putting a human in the loop does not solve everything

The standard answer to AI risk comes in four words: a human will approve.

It is reassuring. It is not always true.

An operator who has to approve forty decisions a day is not examining forty decisions. After a few weeks, they click. The control exists on paper and has disappeared in practice — and it has done something worse, by giving the organisation the feeling of being covered.

This is not an individual failing, it is a design effect. It is well known elsewhere: in internal control, an exhaustive check nobody can sustain is a check that will not be sustained.

Better to separate three things. Approval before the action, which blocks and costs a great deal — to be kept for what is irreversible or heavy. Oversight after the fact, which lets things through but allows correction. And control by exception, where a human steps in only on cases the system itself flags: low confidence, unusual amount, deviation from the expected profile.

The right question is not: do we need a human? It is: at what point does their judgement actually add something?

The hidden cost of autonomy

A chatbot is one request, one model call, one answer.

An agent is a request, a plan, several searches, tool calls, analysis of what comes back, a reassessment, fresh actions. The user sees one interaction. The infrastructure runs thirty.

That gap explains most of the billing surprises on early agentic deployments. Nothing dramatic, but nothing you see coming while you think in cost per request.

And the cost of a request says very little. The useful metric is the cost of a task correctly completed, reruns included.

An agent three times more expensive to run but that takes a case all the way through without intervention can be excellent. A cheap agent that needs two human reruns per case is not, however pretty its billing line.

The right indicator is not the price of a call. It is the price of the useful work that comes out of it.

Not every call needs the best model

In an agentic chain, the steps are not equally hard.

Sorting a request into a category, extracting an amount, checking that a format is valid, choosing a route: these are simple operations, often deterministic. They do not need the most capable model. Some need no model at all.

Reasoning about an ambiguous case does justify one.

Routing between several models according to the difficulty of the step is one of the rare architectural choices that improves cost, latency and stability at once. It only requires having looked at what the agent actually does, step by step — something few teams take the time to do before going live.

Not everything an agent can do is worth having an AI do.

Knowing how to stop an agent

What does the agent do when it no longer knows?

When a tool returns inconsistent data? When it repeats the same action three times without progressing? When the task budget is exceeded? When it meets a case that resembles nothing it was ever evaluated on?

These questions are rarely asked before go-live. They always come up afterwards.

The answers are ordinary mechanisms borrowed from operations: a hard stop beyond a threshold, escalation to a human with the context of what was attempted, a rollback to the previous state where actions are reversible, a switch to the conventional process when the agent leaves its domain. You also need ceilings — on cost, on steps, on time — and a confidence threshold below which the agent says nothing rather than acting.

An agent stuck in a loop is an operations problem before it is an AI problem. And an agent that completes an action it should have refused is a design problem.

Stopping cleanly is part of autonomy.

Measuring what actually counts

The dashboards of early agentic projects measure satisfaction and the perceived quality of answers. Useful, and nowhere near enough.

What needs tracking is less flattering: the share of tasks completed without intervention, how often humans have to take over, the number of steps per task, the average cost of a successful task, the actions that had to be corrected or cancelled, and the situations that triggered an escalation — those above all, because they show where the system's real limits sit.

These figures serve a single question, asked at regular intervals:

On which tasks can we increase autonomy without degrading quality or losing control of costs?

That is the point where experimentation becomes industrialisation.

Autonomy has to be earned

Agents make it possible to automate sequences of work that call for adaptation and decisions. That is exactly what makes putting them into production more demanding than shipping ordinary software.

The right objective is neither to keep a human behind every action, nor to aim for maximum autonomy. It is to find the useful level of autonomy — the one where quality is measurable, cost is proportionate to value, errors are recoverable, and the organisation is able to take back control.

The question is no longer only: what can this agent do? It becomes: what are we willing to entrust to it?

That second question is what separates an impressive demonstration from a system you can actually run in production.

Sources and references

• Sayash Kapoor, Benedikt Stroebl, Zachary S. Siegel, Nitya Nadgir and Arvind Narayanan (Princeton), AI Agents That Matter, 2024 — the gap between benchmark performance and real reliability, and the cost and reproducibility of agent evaluations. arxiv.org/abs/2407.01502
• Anthropic, Demystifying evals for AI agents, 9 January 2026 — evaluating trajectories, the tools called and the actions performed, not the result alone. anthropic.com/engineering/demystifying-evals-for-ai-agents
• World Economic Forum, From chatbots to personal assistants: how governance is key to harnessing the power of AI agents, 11 September 2026 — governance calibrated to the level of autonomy granted. weforum.org
• Gartner, What the 2026 Hype Cycle for Agentic AI Reveals, 15 April 2026 — telling apart what agentic AI delivers today from what remains emerging. gartner.com