How do you secure an LLM application’s infrastructure?

The outcome you should expect

A properly secured LLM stack does not feel like a fortress from the inside. It feels like a normal web application with a few extra choke points. That is the outcome to aim for: the security posture becomes boring and legible, so a new engineer can point at a diagram and say "requests enter here, get checked here, hit the model here, and get checked again on the way out."

Concretely, when the work is done you should be able to answer five questions without opening a codebase. Who can call the model, and with what credential? What is the maximum a single caller can spend or consume in an hour? What data can leave the boundary in a response? What tools or systems can the model reach on its own initiative? And what record exists, six months later, of any given request? Teams that can answer all five have the substance of LLM security. Teams that can only answer the first have bought a product, not a posture.

The second outcome is a shift in where incidents get caught. Before hardening, the failure mode is discovery-by-invoice or discovery-by-customer: someone notices a $40,000 month, or a user screenshots the model reciting another tenant's data. After hardening, the failure mode is a spike on a dashboard and a blocked request in a log. That is not a small difference. It is the difference between an incident and an event.

Expect the first pass to surface things that were never really about the model. A hardcoded provider key in a container image. A vector database reachable on a public IP because the quickstart said so. An IAM role attached to the inference service that inherited broad read access from a template. In practice, most of what gets found in the first security review of an LLM application is ordinary cloud misconfiguration wearing a new hat. The genuinely novel risks — prompt injection through retrieved content, tool-call abuse, training-data leakage through model outputs — are real, but they sit on top of the same infrastructure hygiene that has always mattered.

Expect, too, that you will not eliminate prompt injection. No serious practitioner claims otherwise. The realistic outcome is containment: injection succeeds occasionally, and when it does the blast radius is one user's session with a read-only token and no ability to reach anything that matters. Design for that. The industry framing here comes from the OWASP Top 10 for LLM Applications, which is worth reading end to end before you scope any of this work — it names the categories your threat model should cover and keeps the conversation grounded in something other than vendor marketing.

Finally, expect the secured version to cost measurably more per request and slightly more in latency. A gateway hop, a moderation call, and structured logging together typically add tens of milliseconds and a modest per-request fee. Teams that budget for this up front ship it. Teams that discover it during a launch review cut it.

What drives that outcome

Four forces do most of the work, and they compound. Understanding which one you are short on tells you where the next week of effort goes.

The trust boundary. The single highest-leverage decision is where you draw the line between trusted and untrusted. In a classic web app, user input is untrusted and everything server-side is trusted. LLM applications break that cleanly, because a retrieved document, a tool response, a scraped page, or a PDF a user uploaded all become part of the prompt — and therefore all become instructions the model may follow. The correct posture is that everything entering the context window is untrusted, including content your own systems fetched. This one reframe kills a large class of bugs before they exist, because it forces you to ask what a malicious document could make the model do, rather than assuming your own data is safe.

Credential scope. LLM applications tend to accumulate credentials: a provider API key, a vector database connection, an object store, and increasingly a set of tools the model can invoke. Each is an opportunity to over-grant. The pattern that holds up is per-tenant, per-session, short-lived credentials issued at request time, with the model's effective permissions being the intersection of what the tool allows and what the calling user is allowed to do. A model that can call a "look up customer" function should be doing so through a token minted for the requesting user, not through a service account that can read all customers. When injection eventually succeeds, this is the control that decides whether it is embarrassing or catastrophic.

Rate and cost limits. LLM inference is the rare workload where an attacker's payload costs you real money per token. Denial-of-wallet is a genuine attack, not a theoretical one, and unlike traditional DoS it can be executed slowly and quietly. Limits belong at multiple levels: per API key, per authenticated user, per IP, and per organization, with a hard monthly ceiling that fails closed. Also cap output tokens per request — an unbounded max_tokens with a model that will happily generate until it hits the context limit is a straightforward amplification vector.

Observability. You cannot secure what you cannot see. Structured logging of request metadata — caller identity, model, token counts, latency, tool calls invoked, moderation verdicts — is the substrate everything else builds on. Note the tension: full prompt and completion logging is enormously useful for incident response and enormously risky for privacy. The workable compromise for most teams is metadata always, content selectively, with redaction applied before write and a short retention window on the content tier.

The diagram is deliberately linear, and that linearity is the point. Every control sits on a single path that all traffic must traverse. The most common architectural mistake in LLM security is having two paths — the hardened one the security review saw, and the direct-to-provider one an internal batch job uses because the gateway was inconvenient. Make the secured path the only path by removing provider credentials from application code entirely and holding them exclusively at the gateway.

Benchmarks and realistic ranges

Numbers here vary by provider, region, and model, so treat these as planning ranges rather than guarantees — verify against current published pricing before you commit a budget.

Latency overhead. A well-placed gateway hop adds single-digit to low-double-digit milliseconds. Synchronous moderation or classification calls are the expensive part, because they are themselves model calls: a lightweight classifier typically lands in the tens of milliseconds, while an LLM-as-judge check on the prompt can add several hundred. Against a streaming generation that takes several seconds end to end, a fifty-millisecond input check is invisible to users. A five-hundred-millisecond one is noticeable on the time-to-first-token, which is the latency metric users actually feel. If you need heavyweight semantic checks, run them asynchronously for detection and alerting rather than synchronously for blocking, or reserve them for the subset of requests a cheap heuristic already flagged.

Detection accuracy. Be skeptical of any single number here. Heuristic and regex-based injection detection catches the obvious cases — literal "ignore previous instructions" phrasing and its close variants — and misses paraphrase, encoding tricks, multilingual attacks, and instructions embedded in retrieved documents. Embedding-similarity approaches against a corpus of known attacks do better on variants but degrade against novel techniques. Model-based classifiers, including commercial ones, perform best in published evaluations but are also evaluated by their vendors on their own benchmarks. The defensible planning assumption is that layered detection meaningfully reduces the volume of successful injection while never reaching zero, which is exactly why credential scoping matters more than detection quality.

Cost. Gateway and guardrail services typically price per request or per thousand text units, and at moderate volume the security layer lands well under the inference spend it protects — often a low single-digit percentage. The exception is synchronous LLM-based checking, where you are paying for a second inference on every request; that can approach or exceed the cost of the primary call if the checking model is not substantially cheaper. Self-hosted open-source options trade license cost for operational cost: budget engineering time for the container, the scaling, the upgrades, and the on-call rotation, which for most small teams exceeds the managed-service fee within a year.

Rate limit starting points. For an authenticated consumer-facing product, per-user limits in the range of a few dozen requests per minute and a few thousand per day are usually generous enough not to affect legitimate use while capping abuse. Internal tools can be tighter. Public unauthenticated endpoints should be tighter still, and should have a global circuit breaker that trips well below the level where the monthly bill becomes a problem. Set the ceiling by working backward from the invoice you would be willing to pay, not forward from expected traffic.

Time to implement. A managed gateway in front of an existing application is a day or two of work, most of it spent on the endpoint change and testing. Scoped per-request credentials for tool calls is a genuine architectural change — plan a sprint, sometimes two, because it usually means building a token-minting service you did not previously have. A self-hosted proxy is a day to stand up and considerably longer to run properly. Adding structured audit logging with redaction is roughly a week including the retention and access-control decisions, which are usually the slow part because they need legal or compliance input.

Adjacent workloads. The same reasoning transfers with small modifications. RAG pipelines add an ingestion boundary — every document indexed is a future prompt, so the sanitization question moves upstream to ingest time, and per-tenant isolation in the vector store becomes a hard requirement rather than a nice-to-have. Fine-tuning adds a training-data boundary, where the concern is memorization and leakage of anything sensitive in the corpus. Agentic systems, which is where most teams are heading, multiply the tool-call surface: an agent that can browse, execute code, and write to systems has an attack surface closer to a CI runner than a chatbot, and should be sandboxed accordingly — ephemeral compute, no ambient credentials, network egress allowlisted.

Risks, edge cases, and failure modes

The bypass path. Named above, worth repeating as the number one finding in real reviews. Someone will need to call the provider directly — a batch job, a notebook, an evaluation harness, a vendor integration — and will do it with a key that skips every control you built. The fix is organizational as much as technical: provider keys live in exactly one place, rotation is centralized, and any code path holding a raw provider key is a finding. Egress filtering that blocks direct calls to provider domains from application subnets makes this enforceable rather than aspirational.

Indirect injection through retrieval. The nastiest version of prompt injection does not come from your user. It comes from a document your system retrieved — a support ticket a customer wrote, a web page your crawler indexed, an email in a connected inbox. The instruction sits dormant in your vector store until the right query surfaces it. Defenses: structurally separate retrieved content from instructions in the prompt using clear delimiters and explicit framing, never grant retrieved content the authority to trigger tool calls, sanitize at ingest, and treat any retrieved-content-driven tool invocation as requiring confirmation. This is an unsolved problem in the general case; the goal is reducing what a successful injection can reach.

Output-side leakage. Input filtering gets the attention; output filtering catches the incidents. A model that has been given a system prompt containing credentials, internal URLs, or business logic can be induced to recite it. A RAG system with weak tenant isolation can return another customer's document verbatim. Scan responses for the specific patterns that matter to you — credential formats, internal hostnames, ID formats — and validate structured outputs against a schema before they reach any downstream system. If a tool call's arguments come from model output, validate them as hostilely as you would validate a form submission, because functionally that is what they are.

Over-blocking. The mirror failure. Aggressive filters on a legitimate application produce a steady drip of false positives that users experience as the product being broken. A security tool blocking a doctor's clinical question or a fraud analyst's description of a scam is not a hypothetical. Run new filters in shadow mode first — log the verdict, do not act on it — for at least a week of real traffic, review what would have been blocked, and tune before enforcing. Keep a fast path for reporting false positives, because you will need it.

Logging becoming the breach. Full prompt and completion logs are a concentrated store of whatever your users typed, which in some products is the most sensitive data you hold. Treat the log store with the same access controls as the primary database: encrypted, access-audited, retention-limited, redacted at write time for known sensitive patterns. Several publicized AI incidents have involved the logs rather than the model. This is also where the compliance conversation actually lands — a regulated deployment mostly needs to demonstrate access control and retention discipline over this tier.

Model and dependency supply chain. Downloading model weights or a serving container from a public registry is a supply-chain event. Pin versions, verify checksums, and prefer formats that do not execute arbitrary code on load — the pickle deserialization risk in older serialization formats is a real code-execution path. Scan your Python dependencies too; the AI tooling ecosystem moves fast and has produced its share of CVEs in widely used frameworks. Standard dependency scanning in CI covers most of this and costs nothing to add.

Multi-tenant isolation in the vector store. A quiet, high-severity failure. If tenant filtering happens as a post-retrieval application-layer filter rather than a pre-filter enforced by the store itself, a bug or an injected instruction can return cross-tenant results. Enforce isolation at the storage layer — separate namespaces, collections, or indexes per tenant — so that the isolation does not depend on application code being correct.

Silent stoppage. A filter that fails open and stops filtering is worse than no filter, because you believe you are protected. Every control needs a liveness check: a synthetic malicious request sent on a schedule that should always be blocked, alerting if it ever succeeds. The same applies to your logging pipeline. If the audit log stops receiving events, that must page someone.

Streaming. Token-by-token streaming complicates output filtering, because you are shipping content before you have seen all of it. Options: buffer a window before emitting, run detection on the accumulated stream and terminate mid-response on a hit, or accept the risk for low-sensitivity applications. Whatever you choose, decide deliberately — the common outcome is that output filters were designed for the non-streaming path and quietly do nothing once streaming ships.

A practical rollout plan

Sequence matters more than tool selection. This ordering front-loads the controls with the best ratio of risk reduction to effort.

Week one: inventory and choke point. Find every place your code calls a model provider. Grep for the SDK imports and the environment variable names; you will usually find more than expected. Route all of them through one internal client library or one gateway endpoint. Do not add any filtering yet — the goal is a single path. Simultaneously, audit where provider keys live: repositories, CI variables, container images, developer machines. Rotate anything that has been anywhere it should not have been.

Week two: authentication, quotas, and logging. Require an authenticated identity on every model-facing endpoint. Attach per-user and per-organization rate limits with a global ceiling that fails closed. Emit a structured log line per request with caller, model, token counts, latency, and outcome. This week alone eliminates denial-of-wallet and gives you the data to reason about everything after it. Instrument cost per user immediately; the distribution is always more skewed than anyone predicts, and the top of that distribution is where both your abuse and your best customers live.

Weeks three and four: least privilege. Enumerate every credential the inference path holds and cut each to the minimum. Move tool calls onto short-lived tokens minted per request under the calling user's identity. Put the vector store and any internal service on private networking with explicit allowlists. This is the slowest step and the one with the highest payoff, because it is what bounds the damage of every attack you failed to detect.

Week five: input and output inspection, in shadow mode. Add PII redaction on the input side and DLP scanning on the output side, logging verdicts without enforcing them. Add injection heuristics the same way. Review a week of shadow verdicts against real traffic, tune the thresholds, then enforce — output side first, since output filtering has fewer false-positive consequences and catches the leakage cases that matter most.

Week six: adversarial testing and liveness. Test your own controls. Send injection payloads through the real path and confirm they are caught or contained. Attempt to exceed quotas. Try to make the model recite its system prompt. Whatever gets through becomes the next backlog item. Then automate a subset as a scheduled synthetic check so a regression is detected in hours rather than at the next review. Wire alerts to a channel a human actually reads.

Ongoing. Rotate credentials on a schedule. Re-run adversarial tests after every prompt or model change, because both alter behavior in ways that invalidate prior testing — a model upgrade can silently change how the system responds to instructions embedded in retrieved content. Review the top cost consumers weekly. Re-read the OWASP LLM Top 10 when it updates, and map your controls against it rather than against a vendor's feature matrix.

A note on build versus buy. Managed gateways get you weeks two and five quickly and are the right default for small teams. Self-hosted proxies make sense when data residency or audit requirements make a third-party hop unacceptable. Framework-level guardrail libraries fit teams already building custom orchestration who want programmable policy rather than a network hop. None of them do week three for you — least privilege is your architecture, not a product feature — which is why the buy decision matters less than the sequencing.

Related questions

Does a gateway replace application-level authorization?

No. A gateway enforces who may call the model and how much. It has no idea whether this user should see that customer record. Authorization stays in your application and in the credentials you mint for tool calls.

How does securing a RAG pipeline differ?

The trust boundary moves to ingestion. Every indexed document is a future prompt, so sanitize at ingest, enforce tenant isolation in the vector store itself rather than in application filtering, and never let retrieved content authorize a tool call.

Is self-hosting the model more secure?

It removes the third-party data-handling question and adds everything else: GPU infrastructure, patching, serving-stack CVEs, and weight supply chain. More control, more surface. Choose it for data residency requirements, not for a general assumption of safety.

What should be logged, and for how long?

Metadata — identity, model, token counts, latency, tool calls, filter verdicts — always, with long retention since it is low-risk. Prompt and completion content selectively, redacted, with short retention and access controls matching your primary database.

Do agentic applications need different controls?

Yes, substantially. An agent with browsing, code execution, and write access resembles a CI runner. Sandbox in ephemeral compute, allowlist network egress, remove ambient credentials, and require confirmation for irreversible actions.

FAQ

Can any tool stop all prompt injection?

No, and treat claims otherwise as marketing. Detection reduces volume; it does not reach zero, because the attack surface is natural language and paraphrase space is unbounded. The durable defense is containment — scope credentials so a successful injection reaches nothing consequential. Build assuming some attempts will succeed, then make success cheap.

What is the single highest-value control if I can only do one thing?

Authenticated per-user rate and cost limits at a single choke point. It eliminates denial-of-wallet outright, gives you per-user visibility that makes every subsequent control easier to target, and takes a day or two. Least privilege on tool credentials is a close second and matters more long-term, but takes weeks.

Where should provider API keys live?

In exactly one place — a gateway or a secrets manager the gateway reads from — never in application code, container images, or client bundles. Application services authenticate to your gateway with their own credentials. Block direct egress to provider domains from application subnets so the rule is enforced rather than documented.

How do I handle compliance requirements like HIPAA or GDPR?

Most of what auditors want is not LLM-specific: encryption in transit and at rest, access control, audit logging with defined retention, and a data processing agreement with any provider that sees regulated data. The LLM-specific additions are output DLP to prevent regulated data leaving in responses, and a documented position on whether prompts are used for training — check the provider's enterprise terms.

Does adding all this security break my latency budget?

A gateway hop and lightweight filtering add tens of milliseconds, which is negligible against multi-second generation. Synchronous LLM-based checking is what hurts, adding hundreds of milliseconds to time-to-first-token. Run heavyweight semantic checks asynchronously for alerting, or only on requests a cheap heuristic already flagged.

How often should controls be retested?

After every model change, prompt change, or tool addition, since all three alter behavior in ways that invalidate prior results. Automate a small suite of adversarial payloads as a scheduled synthetic check so regressions surface in hours. Do a deeper manual adversarial pass quarterly, or before any significant launch.

Sources

• OWASP Top 10 for Large Language Model Applications
• NIST AI Risk Management Framework
• MITRE ATLAS — Adversarial Threat Landscape for AI Systems
• Cloudflare AI Gateway documentation
• Amazon Bedrock Guardrails
• Azure AI Content Safety documentation
• OpenAI Moderation API guide
• NVIDIA NeMo Guardrails on GitHub
• Google Cloud Armor documentation
• CISA Guidelines for Secure AI System Development

Related on PULSE

• [How do you set up observability for a RAG application?](/knowledge/ai387)
• [What infrastructure do you need to run AI agents in production?](/knowledge/ai373)
• [What infrastructure do you need for fine-tuning versus RAG?](/knowledge/ai427)
• [What is the difference between batch and real-time inference infrastructure?](/knowledge/ai409)
• [What is the role of Kubernetes in modern AI infrastructure?](/knowledge/ai431)