What infrastructure do you need to run AI agents in production?

What an agent stack actually is, and why it differs from a normal app

An AI agent is not a single API call. It is a loop: the model receives context, decides whether to answer or call a tool, the runtime executes that tool, the result is appended to context, and the loop repeats until a stopping condition. That loop shape is the whole reason agent infrastructure looks different from the infrastructure behind a conventional web service, and it is why teams who treat an agent like a stateless HTTP endpoint get surprised in week three.

Consider what the loop implies. A normal request/response service has a bounded, predictable cost per request — you can load-test it, find the p99, and provision against that number. An agent's cost per request is a distribution with a long tail. One user question resolves in a single model call and 900 tokens. Another triggers seven tool calls, drags 40,000 tokens of retrieved documents into context, and runs for ninety seconds. The variance between those two is often 50x in cost and 100x in wall-clock time, and both arrive at the same endpoint with no way to tell them apart in advance. Every piece of infrastructure you build has to tolerate that spread.

The second structural difference is that context grows within a single logical request. In a normal service, the payload is fixed at the door. In an agent, each tool result gets appended to the conversation, so the tenth model call in a loop is processing far more tokens than the first. Since most inference pricing is per-token and most latency scales with context length, a long agent trajectory is superlinear in both cost and time. This is the single most common reason a demo that felt instant becomes a production system that feels sluggish — nothing broke, the loop just got longer.

Third, agents have side effects. A retrieval-augmented chatbot reads. An agent writes: it files tickets, sends emails, updates CRM records, refunds orders, kicks off deploys. That moves the infrastructure conversation from "is it fast enough" to "what happens when it does the wrong thing twice." You need idempotency keys on tool calls, an audit trail of every action with the reasoning that produced it, and a kill switch that stops the fleet without a redeploy. These are not AI problems — they are the same problems distributed-systems and payments engineers have solved for two decades — but agent teams routinely rediscover them the hard way because the prototype had no side effects at all.

The fourth difference is nondeterminism. The same input can produce different tool-call sequences on different runs, even at temperature zero, because of batching effects, model version drift, and retrieval index changes. That breaks the usual testing contract. You cannot assert exact output equality; you have to assert on properties — did it call the right tool, did it stay within the token budget, did it refuse when it should have. This shapes your CI infrastructure as much as your runtime.

Put together, the practical requirement list is: something that serves the model, something that runs the loop, somewhere to keep state between turns, a control point in front of it all, and enough telemetry to reconstruct any given run after the fact. The rest of this page walks each layer, then the money, then the failure modes.

The step-by-step process for standing one up

Here is the sequence that works, in order, with what each step actually involves.

Start with hosted inference, not GPUs. Nearly every team that buys or rents GPUs on day one regrets it. Hosted model APIs from the major providers give you sub-second time-to-first-token, no capacity planning, and per-token billing that scales to zero. You will not know your real token volume, model size requirement, or latency budget until you have run production traffic. Buying dedicated capacity before you have that data means you are optimizing a number you have not measured. The exception is a genuine regulatory constraint — data that cannot leave your network — in which case self-hosting is a requirement, not an optimization.

Pick an orchestration runtime and treat it as a durable workflow problem. The loop needs to survive process restarts. If your agent is fifteen tool calls deep and the container gets rescheduled, you want it to resume, not restart from zero and re-execute the six side-effecting calls it already made. Durable execution engines — the workflow-orchestration category, or a homegrown state machine backed by a database — solve exactly this. The naive version, a while loop inside a request handler, works fine until your first deploy during business hours.

Wire in state before you need it. Agents need three distinct kinds of storage and teams tend to conflate them. Short-term conversation state (the messages in the current session) belongs in a fast key-value store with a TTL. Long-term memory (facts about the user, prior resolutions) belongs in a durable database, usually relational, because you will need to query and correct it. Retrieval corpora belong in a vector index. These have different access patterns, different consistency requirements, and different costs. Using your vector database as a session store is a common and expensive mistake.

Put a gateway in front of everything. One service that all agent traffic passes through, which handles: authentication and tenant identification, per-tenant rate limits, per-request and per-tenant token budgets with hard cutoffs, model routing (cheap model for classification, expensive model for reasoning), retry logic with backoff, and response caching where the input is genuinely repeated. Building this yourself is a week of work; several open-source and commercial LLM gateways do it off the shelf. Without it, you will discover your cost problem from a bill rather than from a dashboard.

Instrument every step, not every request. Standard APM tells you the request took 40 seconds. That is useless. You need a trace where each span is one model call or one tool execution, tagged with input tokens, output tokens, model version, latency, tool name, and outcome. OpenTelemetry has semantic conventions for exactly this now, and the LLM-observability vendors all speak it. The test of whether your instrumentation is adequate: when a user complains about a bad answer, can you pull up that exact run and read the full trajectory in under two minutes? If not, you are debugging blind.

Build the evaluation harness before you build features two through ten. A set of recorded inputs with expected properties, run on every change to a prompt, model version, or tool definition. It does not need to be sophisticated — fifty cases and a handful of assertions catches the majority of regressions. What it needs to be is automatic, because prompt changes feel free and are not.

Deploy behind a flag with a blast radius you can tolerate. Percentage rollout by tenant, a global disable switch that takes effect without a deploy, and for any agent with write permissions, a human-approval step on the highest-consequence tools until you have weeks of clean traces. Approval queues feel like a step backward from full autonomy. They are how you earn the right to remove them.

Costs, timelines, and typical ranges

Money is where agent projects surprise people, so let us be concrete about the shape of it, while staying honest about the fact that vendor pricing moves constantly and you should check current rate cards rather than trust any number written down months ago.

Token cost is the dominant line item for most teams on hosted inference, and it is driven by context length more than by request count. The arithmetic that matters: total cost ≈ (average tokens per model call) × (average model calls per agent run) × (runs per month) × (per-token rate). Teams estimate the first and third terms and forget the second. An agent averaging six model calls per run, each carrying an accumulated context, can easily consume ten to twenty times the tokens of a single-shot chat completion for the same user question. When you model your budget, model the trajectory, not the request.

The biggest available cost lever is prompt caching, and it is underused. Agent loops re-send the same system prompt, tool definitions, and retrieved documents on every iteration. Providers that support caching that prefix charge substantially less for the cached portion on subsequent reads. Because agent loops are structurally repetitive, the hit rate can be very high if you order your context correctly — stable content first, volatile content last. Restructuring context to be cache-friendly is often the highest-return afternoon of work available on an agent codebase.

The second lever is model routing. Not every step in a trajectory needs your most capable model. Intent classification, extraction of structured fields from a tool response, deciding whether retrieved context is relevant — these are small-model tasks. Reserve the frontier model for the reasoning and synthesis steps. A well-routed stack commonly cuts spend meaningfully without a measurable quality difference, and the routing logic lives naturally in the gateway you already built.

Self-hosted GPU economics have a clear crossover point. Renting GPU capacity by the hour and running it at low utilization is more expensive than hosted APIs; the same hardware at high sustained utilization is cheaper. The break-even depends on your token volume, the model size, and how good your batching is. The rule of thumb that holds: if your GPU utilization would sit below roughly half, hosted inference wins on cost and wins massively on operational burden. Serverless GPU platforms sit in between — per-second billing with cold-start latency measured in seconds, which suits bursty batch workloads and does not suit interactive agents.

The infrastructure around the model is cheaper than people fear and more expensive than people budget. Vector databases, key-value stores, workflow engines, and observability tooling are ordinary cloud spend at ordinary cloud prices — a rounding error against inference at low volume, a real line item at high volume, and never the thing that blows up a budget. What blows up budgets is unbounded loops, missing token caps, and a retrieval step that pulls fifty documents when five would do.

Timelines, based on the shape the work usually takes. A working prototype against a hosted API: days. A version with orchestration, state, a gateway, and traces, running for internal users: several weeks. A version with evaluation coverage, incident runbooks, cost controls, tenant isolation, and enough production data to trust it externally: a quarter, give or take, and most of that time goes into the boring parts — permissions, error handling, the long tail of tool failures — rather than into prompts. Teams that scope only the prototype and treat the rest as polish ship late.

Where the hidden costs live. Retries against a flaky tool that each carry a full model call. Debugging time lost because traces were sampled and the bad run was not captured. Re-embedding an entire corpus after an embedding model upgrade. Human review labor on approval queues that nobody budgeted headcount for. Egress on moving retrieval corpora between clouds. None of these are exotic; all of them are invisible in a proof of concept.

Where teams get it wrong

No step limit, no budget cap. The prototype has neither because the prototype never loops more than three times. Production has a user whose phrasing sends the agent into a tool-calling cycle that runs until something times out. Hard caps on steps per run, tokens per run, and spend per tenant per day are not optimizations — they are the guardrails that keep one bad input from becoming an incident. Cap first, tune later.

Treating retrieval as solved because the demo worked. Retrieval quality dominates agent quality more than model choice does, and the demo corpus was small, clean, and hand-picked. Production corpora are large, stale in places, contradictory in others, and full of near-duplicate documents that crowd out the one useful result. Chunking strategy, hybrid keyword-plus-vector search, reranking, and freshness handling all matter. If your agent hallucinates, look at what you retrieved before you look at the prompt — in most investigations, the model faithfully summarized bad context.

Giving the agent broad credentials. The service account is provisioned with wide permissions because narrowing it was fiddly during development. Now a prompt-injected instruction inside a retrieved document, or a customer email the agent reads, executes with those permissions. Scope every tool to the minimum it needs, scope tool access to the calling user's own permissions rather than a shared superuser, and treat any content the agent reads from outside your trust boundary as untrusted input — because it is. The published guidance on LLM application security converges on this point.

No idempotency on write tools. The run gets retried after a timeout. The refund is issued twice. Every side-effecting tool needs a caller-supplied idempotency key derived from the run ID and step index, and the downstream system needs to honor it. This is standard practice in payments and it applies unchanged here.

Sampling traces. APM defaults sample at low percentages to control cost. Agent traces are the primary debugging artifact and the raw material for your evaluation set. Capture all of them, at least at first, and set retention rather than sampling as the cost control. The run you failed to record is always the one the customer is asking about.

Pinning nothing. Model versions change, embedding models change, tool APIs change. If you are calling a floating model alias, your behavior can shift without a deploy on your side. Pin explicit versions, subscribe to deprecation notices, and re-run your eval suite against a new version in staging before you move production traffic.

Assuming average latency is the user experience. Agents have heavy tails. The p50 is fine and the p95 is a user staring at a spinner for forty seconds. Stream partial output, show which tool is executing, and set an explicit deadline after which the agent returns what it has with an honest explanation. Perceived latency is an infrastructure problem you solve at the transport layer, not a prompt problem.

Skipping evaluation because the outputs are subjective. They are partly subjective and largely not. Did it call the right tool? Stay under budget? Refuse the out-of-scope request? Cite a document that actually contains the claim? Those are all mechanically checkable. Start with the checkable ones and add human or model-graded judgment later.

Underestimating the tool layer. Most production agent failures are not model failures. They are a tool returning a 500, a schema that drifted, a timeout with no retry, or a response so large it blows the context window. Your tool implementations need the same defensive engineering as any integration layer: timeouts, retries with backoff, schema validation on responses, and truncation of oversized payloads before they reach the model.

Decision framework: when to choose what

The choices that matter are fewer than the vendor landscape suggests. Work through them in this order.

Can your data leave your network? If a regulatory or contractual constraint says no, self-hosting is settled and the rest of the decision tree is about which open-weight model and which serving stack. If yes, hosted inference is the default and you need a specific reason to move off it.

Is the workload interactive or batch? Interactive agents — a user waiting — need consistently low time-to-first-token, which rules out cold-start-heavy serverless GPU for the hot path. Batch agents — overnight enrichment, document processing, bulk classification — can tolerate cold starts and preemption, which unlocks spot pricing and serverless per-second billing at a meaningful discount. Many organizations run both, on deliberately different infrastructure, and that is the correct answer rather than a failure to standardize.

What is your sustained utilization? Low and spiky argues for hosted APIs or serverless. High and steady argues for reserved or dedicated capacity, whether that is provisioned throughput from a hosted provider or hardware you manage. Provisioned throughput deserves a specific mention: several hosted providers sell guaranteed capacity units, which give you predictable latency and predictable cost at the price of committing ahead of demand. That is often the right middle rung — dedicated performance without becoming a GPU operations team.

How many agents talk to each other? A single agent with tools is one problem. A multi-agent system where a supervisor delegates to specialists is a distributed system, and it needs the corresponding infrastructure: a message bus or workflow engine for coordination, shared state with a clear ownership model, distributed tracing that stitches child runs to parent runs, and a global budget that no individual agent can blow through. Do not build multi-agent until a single agent with more tools has demonstrably failed — the coordination overhead is real and the debugging difficulty compounds.

Build or buy the middle layer? Orchestration, gateway, and observability all have credible commercial and open-source options. The honest guidance: buy observability, because building good tracing is a lot of undifferentiated work and the vendors are ahead of you. Buy or adopt an existing gateway for the same reason. Be more willing to own orchestration, because your agent's control flow is where your actual product logic lives and framework abstractions tend to fight you exactly when the requirements get specific. A thin, explicit loop you wrote, backed by a durable execution engine, ages better than a deep framework you did not.

Adjacent workloads that reuse the same stack. Once this infrastructure exists, it serves more than agents. RAG chat, document classification pipelines, structured extraction from contracts or invoices, code review bots, internal search — all of them want the same gateway, the same tracing, the same eval harness, and the same secret management. The marginal cost of the second AI product on this platform is a fraction of the first, which is a genuine argument for building it properly rather than embedding it inside one application. It is the same platform-team logic that justified centralized CI or a shared feature store, and it plays out the same way: the second and third consumers are what make the investment obviously correct in hindsight.

Related questions

Do I need Kubernetes to run agents in production?

No. Kubernetes helps when you self-host models or run many long-lived services, but agents on hosted inference run fine on serverless functions, containers, or a managed platform. Choose Kubernetes because your organization already operates it, not because agents require it.

How much GPU memory does a self-hosted model need?

Roughly two bytes per parameter at 16-bit precision, plus headroom for the KV cache, which grows with context length and concurrency. Quantization to 8-bit or 4-bit cuts the weight footprint substantially at some quality cost. Benchmark your own model and context length rather than trusting a table.

Can agents run entirely on CPU?

For small models and low throughput, yes — CPU inference works and costs less. It is typically far slower per token than GPU, which makes it unsuitable for interactive agents where a multi-step loop compounds the delay. CPU is reasonable for offline batch jobs with generous deadlines.

What is the difference between an LLM gateway and an API gateway?

A conventional API gateway handles auth, routing, and rate limits by request count. An LLM gateway adds token-aware budgets, model routing and fallback, prompt caching, semantic response caching, and per-model cost attribution. You often want both, with the LLM gateway sitting behind the general one.

How do I keep an agent from taking a destructive action?

Scope tool permissions narrowly, require explicit approval on high-consequence tools, make writes idempotent, keep a full audit trail linking action to reasoning, and maintain a runtime kill switch. Treat autonomy as something granted incrementally after clean production traces, not as the starting configuration.

FAQ

What is the smallest viable production stack for a single agent?

A hosted model API, a stateless service running the agent loop, a managed relational database for conversation and memory state, a managed vector store if you need retrieval, and an observability tool that captures per-step traces. That is five components, all managed, and it will carry meaningful production traffic. Add durable workflow execution the first time a mid-run restart causes a duplicate side effect, and add a gateway the first time cost or rate limits become a real concern. Building all of it up front is a common way to spend a quarter without shipping.

When should we move from hosted APIs to our own GPUs?

When one of three things is true: a compliance requirement forbids sending data to a third party, your sustained utilization is high enough that dedicated capacity is cheaper than per-token pricing, or you need a fine-tuned or otherwise custom model that no hosted provider serves. Absent one of those, self-hosting adds capacity planning, driver and kernel maintenance, batching optimization, and on-call burden without a corresponding benefit. Run the utilization math with real production numbers before committing.

How do we test agents when the output is nondeterministic?

Assert on properties rather than exact strings. Build a fixture set of recorded inputs and check mechanically verifiable things: which tools were called and in what order, whether token and step budgets were respected, whether required refusals happened, whether cited sources actually contain the cited claim. Layer model-graded or human evaluation on top for the genuinely subjective dimensions. Run the whole suite on every prompt, model version, and tool schema change, and gate deploys on it.

What observability signals matter most for agents in production?

Per-step traces with token counts and model versions, cost per run broken down by tenant and by workflow, distribution of steps per run (a rising tail means looping behavior), tool error rates by tool, time-to-first-token separate from total latency, and rates of refusals, timeouts, and budget-cap hits. The distribution of steps per run is the most underrated of these — it moves before your bill does and is the earliest warning that behavior has drifted.

How should we store agent memory?

Split it by lifecycle. Session state — the current conversation — goes in a fast key-value store with a TTL matching your session window. Durable facts about users or accounts go in a relational database where you can query, correct, and delete them, which also handles data-subject deletion requests cleanly. Retrieval corpora go in a vector index. Keeping these separate makes each one cheap, debuggable, and independently correctable; conflating them produces a store that is expensive to query and impossible to audit.

Does the same infrastructure support multiple AI products?

Yes, and that is a strong argument for building it deliberately. The gateway, tracing, evaluation harness, secret management, and retrieval layer are shared across agents, RAG chat, classification pipelines, and extraction jobs. Teams that build this as a platform find the second and third AI product take a fraction of the time of the first. Teams that embed it inside one application end up rebuilding it, slightly differently, for every subsequent use case.

Sources

• OpenTelemetry semantic conventions for generative AI
• OWASP Top 10 for Large Language Model Applications
• NIST AI Risk Management Framework
• Anthropic docs: tool use with the Claude API
• Anthropic docs: prompt caching
• Google Cloud: Vertex AI Agent Builder documentation
• AWS documentation: Amazon Bedrock Agents
• Microsoft Learn: Azure AI Foundry documentation
• Temporal documentation: durable execution concepts
• Kubernetes documentation: scheduling GPUs

Related on PULSE

• [How do you handle model rollbacks safely in production?](/knowledge/ai429)
• [How do you A/B test different LLMs in production?](/knowledge/ai391)
• [How do you choose a vector database for a production RAG system in 2027?](/knowledge/ai339)
• [How do you monitor LLMs in production for drift and hallucinations?](/knowledge/ai355)
• [How do you secure an LLM application's infrastructure?](/knowledge/ai363)
• [What infrastructure do you need for fine-tuning versus RAG?](/knowledge/ai427)