What it takes to run AI agents in production

An agent demo takes an afternoon. An agent you'd let touch a customer record takes considerably longer, and almost all of the extra work is in places the demo never goes: what happens when the model is confidently wrong, when a tool call fails halfway, when the provider is down, and when someone asks six months later why the system did a particular thing.

This is what we've found actually has to exist before an agent is safe to run in production.

Define the boundary before the behaviour

The first design question isn't what the agent should do. It's what it must never do without a human.

Write that boundary down explicitly and enforce it in code, not in the prompt. A prompt instruction not to issue refunds is a suggestion; a tool layer that has no refund capability is a guarantee. The distinction matters because prompts get edited, models get swapped, and inputs get adversarial — while the absence of a function stays absent.

Tools are an API design problem

The quality of an agent is mostly the quality of the tools you give it. Teams that struggle usually exposed their internal API surface directly and hoped the model would figure it out.

A few things consistently help:

• Make tools task-shaped, not endpoint-shaped. find_customer_by_email beats a generic query tool that takes arbitrary filters. Narrower tools are easier for the model to select correctly and far easier to validate.
• Return errors the model can act on. "No customer found with that email — try search_customers_by_name" produces recovery. A 404 produces a retry loop.
• Make every tool idempotent where you can. Agents retry. If a retried call can double-charge someone, that's a design fault, not a model fault.
• Bound the outputs. A tool that can return ten thousand rows will eventually return ten thousand rows into a context window. Paginate and cap at the tool layer.

Retrieval: grounding beats instructing

Telling a model not to make things up doesn't work reliably. Giving it the actual text and requiring it to cite which passage it used works considerably better, because a claim with no supporting passage is now detectable.

The parts of a RAG pipeline that turn out to matter most are rarely the parts that get discussed:

• Chunking that respects document structure. Splitting on a fixed token count cuts tables in half and separates headings from the content they govern. Split on sections.
• Metadata filters before semantic search. Most real questions have a hard scope — this customer, this year, this product. Filter first, then rank. It's faster and dramatically more accurate.
• A "not in the documents" path. The system needs an explicit way to say it doesn't know. Without one, the model will fill the gap, and it will sound convincing while doing it.

Evaluation is the part that makes it engineering

This is the single biggest difference between an agent that ships and one that stays in prototype. Without an evaluation set, every prompt change is a guess and every model upgrade is a gamble.

It doesn't need to be sophisticated to be useful. Start with:

1. Thirty to fifty real cases with known-correct outcomes, drawn from actual historical work rather than invented.
2. A scoring function per case. Exact match where the answer is structured; a rubric where it isn't.
3. A regression run on every prompt, tool or model change — in CI, not manually.
4. Deliberate failure cases: missing data, ambiguous requests, and inputs that try to talk the agent out of its instructions.

The first time you run a real evaluation set against a prompt everyone was happy with is usually instructive. Scores of sixty to seventy percent on something that felt like it worked are entirely normal.

Cost and latency are architecture, not billing

Agent costs scale with steps, and steps multiply quietly. An agent that averages six tool calls per task at ten thousand tokens of context each is a different economic proposition from a single completion — and nobody notices until the first full month's invoice.

What keeps it under control:

• Route by difficulty. Classification and extraction rarely need your most capable model. Reserve it for the reasoning step that actually needs it.
• Cache aggressively. Identical retrievals and repeated system context shouldn't be paid for twice.
• Cap the loop. A hard maximum on steps per task. Agents that can loop indefinitely eventually will.
• Budget per task and alert on outliers. A task costing ten times the median is a bug worth seeing before month end.

Design for the provider being down

Model providers have outages, rate limits and deprecation schedules. Building against one provider's SDK throughout your codebase converts each of those into an incident.

Keep the provider behind an interface of your own. Two benefits follow: you can fail over when a provider degrades, and you can evaluate a competing model against your own test set without a rewrite. We build every agent this way, and it has never once been the thing we regretted.

Observability, because someone will ask

At some point — often months later, often from a regulator, a customer or a lawyer — someone will ask why the system did a specific thing on a specific date. "The model decided" is not an answer anyone accepts.

Log, per task: the input, retrieved context with source identifiers, each tool call with arguments and result, the model and prompt version, token counts and cost, the final output, and whether a human approved or edited it. Store it somewhere queryable and keep it as long as the domain requires.

This is also, incidentally, how you improve the system. Every human edit in the approval queue is a labelled correction. Feed those back into the evaluation set and the agent gets measurably better at exactly the cases it currently gets wrong.

The honest summary

The model is the easy part now, and it gets easier every few months. What separates an agent that runs in production from one that doesn't is unglamorous: bounded tools, grounded retrieval, a real evaluation set, cost controls, provider independence and a full audit trail.

None of that is novel software engineering. It's the same discipline any system handling real work has always needed — applied to a component that happens to be probabilistic, and which will confidently tell you it's right when it isn't.

We've written up one of these in detail, including the approval-gate design, in the AI sales automation case study.