What a production AI agent actually needs

Building an agent that works in a demo takes an afternoon. Building one that a finance or support team can rely on every day is a different job. The model is rarely the hard part — the hard part is everything around it.

This is the checklist we walk through before an agent touches real systems.

1. A job description, not a personality

Most failed agents were never told what their job was. "You are a helpful assistant for operations" is not a scope.

Write the agent's job the way you would brief a new hire:

• Inputs — what starts a run (an email, a ticket, a schedule) and what a valid input looks like.
• Done — what a finished run produces, and for whom.
• Out of scope — what it must never attempt, even if asked.
• Escalation — when it should stop and hand the case to a person.

A tight scope makes everything downstream easier: smaller tool sets, simpler evals, fewer surprises.

2. Tools with the least privilege

An agent is only as dangerous as the tools you give it. Each tool should do one thing, take typed inputs, and validate them before touching your systems.

Split reads from writes, and mark anything with side effects explicitly:

const tools = {
  // Read-only: safe to call autonomously.
  getInvoice: tool({
    input: z.object({ invoiceId: z.string().regex(/^INV-\d{6}$/) }),
    run: ({ invoiceId }) => erp.invoices.get(invoiceId),
  }),
  // Side effect: money moves, so a person approves first.
  issueCreditNote: tool({
    input: z.object({ invoiceId: z.string(), amount: z.number().positive().max(5_000) }),
    requiresApproval: true,
    run: (args) => erp.creditNotes.create(args),
  }),
};

Give the agent its own service account with only the permissions those tools need. If a tool can delete records, ask whether it really needs to.

3. Approval gates for anything irreversible

Autonomy is a dial, not a switch. Drafting a summary can run on its own; refunding a customer, emailing a client or changing a contract should wait for a human — at least until the agent has earned trust on real traffic.

Good approval gates are fast: the approver sees what the agent wants to do, why, and the evidence, and can approve, edit or reject in one click from wherever they already work (Slack, email, the ticketing tool).

4. Evals before every release

"It looked right when I tried it" is not a test. Before launch, collect real cases — including the awkward ones — and turn them into an eval set with expected outcomes.

Then run it on every change: a new prompt, a new model version, a new tool. Track the results over time. When a model provider ships an update, your eval set tells you in minutes whether your agent got better or worse.

Useful checks include:

• Did it pick the right tools, in a sensible order?
• Is the output in the expected structure?
• Does it respect business rules (limits, policies, tone)?
• Does it escalate the cases it should?

5. Tracing you can replay

When something goes wrong — and eventually something will — you need to answer what happened without guessing. Log every run end to end: the input, each model call, each tool call with arguments and results, the final output, latency and cost.

Traces are also your best source of new eval cases. Every confusing run is a test you didn't have yet.

6. Budgets and limits

Agents loop. Put hard limits on everything that can run away:

• maximum steps and tool calls per run,
• token and time limits per run,
• spend caps per agent and per month, with alerts before you hit them.

A run that hits a limit should stop cleanly and escalate — not keep trying.

7. A way back

Version your prompts, tool definitions and model choices like code, and deploy them the same way. Keep a kill switch that pauses the agent and routes work back to the human queue. If a release misbehaves, rolling back should take minutes.

8. An owner

Every production agent needs a named owner on the business side and one on the technical side. Someone reviews the escalations, watches quality and cost, and decides when the agent gets more autonomy — or less.

The short version

Before an agent goes live, you should be able to say yes to all of these:

1. Its scope, inputs, outputs and escalation rules are written down.
2. Every tool is narrow, validated and least-privilege.
3. Irreversible actions require approval.
4. An eval set of real cases runs on every change.
5. Every run is traced and replayable.
6. Steps, time and spend are capped.
7. You can roll back or pause it in minutes.
8. It has an owner.

None of this is glamorous, and all of it is what separates an agent people trust from one they quietly stop using.