How do you route requests across multiple LLM providers?

What it is and why it matters

At its core, provider routing solves a dependency problem: any single LLM API can go down, rate-limit you, or get deprecated, and if your application calls that provider directly, your entire product goes down with it. A router sits between your code and the providers, normalizing the request/response shape (prompt in, completion out, regardless of vendor) so your application code stops caring which company is actually serving the request. You send one request; the router decides which provider handles it and returns a response in a consistent format.

This matters for three overlapping reasons. First, resilience — providers have outages, and a router with automatic failover keeps your product answering questions even when your primary vendor is degraded. Second, cost — different providers price the same class of task very differently, and a router lets you shift volume toward whichever provider is cheapest for a given task without rewriting integration code. Third, leverage — once your application no longer speaks a single vendor's proprietary SDK, you can renegotiate, swap, or multi-source providers as pricing and quality shift, which matters more every quarter as new frontier and open-weight models launch.

The practical unit of work here is the "request" — a single call carrying a prompt, parameters, and (usually) a target model name. A router's job is to take that one request and, transparently to the caller, decide which of several possible providers should actually execute it. That decision can be static (you always route model X to provider Y), rule-based (route by task type, region, or customer tier), or dynamic (route by real-time price, latency, or health signals). Most production setups blend all three: a default mapping for normal operation, with fallback and load-balancing rules that kick in only when something goes wrong or a budget threshold is crossed.

Two architectural patterns dominate. The first is the hosted gateway — you send requests to a single vendor-neutral endpoint (OpenRouter and Portkey both work this way), and that vendor's infrastructure holds your provider credentials, applies routing logic, and bills you either per-token or per-request on top of the underlying model cost. The second is the self-hosted proxy — libraries like LiteLLM or LangChain's model abstractions run inside your own infrastructure, hold your own API keys for each provider, and apply routing logic in code you control. Hosted gateways trade a small amount of per-token markup and reduced data control for zero infrastructure burden; self-hosted proxies trade setup and maintenance time for full control over data residency, credentials, and routing logic.

The step-by-step process

Building a multi-provider router follows roughly the same sequence regardless of whether you buy a hosted gateway or build your own with an open-source library.

1. Normalize the interface. Pick a canonical request/response schema (most teams converge on something close to OpenAI's chat-completion shape, since it's the most widely supported) and translate every provider's API into and out of that shape. This is the step hosted gateways and libraries like LiteLLM do for you; if you build it yourself, expect to write an adapter per provider.
2. Register providers and credentials. Each provider — OpenAI, Anthropic, Google, Meta (via a hosting partner), Mistral, and so on — needs its own API key, base URL, and rate-limit ceiling stored in your router's configuration. Keep these in a secrets manager, not in application code, since a router is the single place that now holds every provider credential your product depends on.
3. Define the primary routing rule. Decide, for each logical "model slot" in your product, which provider serves it by default. This is usually a straightforward mapping: your "fast/cheap" slot points at a smaller model, your "high-quality" slot points at a frontier model.
4. Define fallback order. For each primary, specify one or two backups the router should try if the primary errors, times out, or returns a rate-limit response. A common pattern is a same-capability fallback (frontier model A fails over to frontier model B) rather than silently downgrading a user to a much weaker model.
5. Set timeout and retry limits. A request that hangs for 30+ seconds is often worse than one that fails fast and reroutes. Most production routers set a per-attempt timeout in the 8-20 second range with one or two retries before failing over, and a hard ceiling (often 30-45 seconds total) after which the router gives up and returns an error rather than leaving a user staring at a spinner.
6. Add health and rate-limit awareness. The router should track recent error rates and 429 (rate-limit) responses per provider and temporarily deprioritize a provider that's failing, rather than retrying it request after request.
7. Instrument and observe. Log which provider actually served each request, its latency, its token counts, and its cost. Without this, you can't tell whether your fallback logic is firing correctly or whether one provider is quietly eating your budget.
8. Test the failure path deliberately. Before shipping, force a primary-provider failure (an invalid key, a wrong endpoint) in a staging environment and confirm the fallback actually engages and returns a usable response, not just that it "should" based on the config.

Costs, timelines, and typical ranges

The cost of routing itself is usually small relative to the cost of the underlying model calls, but it isn't zero, and it shows up in two places: the router's own fee (if hosted) and the operational cost of running the logic (if self-hosted).

Hosted gateways typically monetize in one of two ways: a thin markup added on top of the underlying provider's per-token price, or a tiered subscription that unlocks features like advanced observability, team management, and higher request volume, with a free or low-volume tier for testing. Self-hosted libraries like LiteLLM or LangChain's routing utilities are free to use under open-source licenses, but you pay in engineering time and in whatever compute you use to run the proxy itself — for most teams that's a small always-on service, often costing a few dollars a month in compute if it's just forwarding requests, since the work is mostly network I/O rather than CPU-bound.

Underlying model costs vary far more than routing costs do, and this is usually the bigger lever. Smaller, faster models generally cost a small fraction of a cent per thousand tokens on the input side, while frontier reasoning models can cost an order of magnitude more per token, with output tokens consistently priced higher than input tokens across every major provider because generation is the more compute-intensive direction. A router that shifts a meaningful share of traffic from a frontier model to a smaller model for tasks that don't need frontier quality — simple classification, short extraction, routine formatting — can cut model spend substantially without any change to the application layer, which is one of the strongest practical arguments for routing in the first place.

Setup timelines depend heavily on which path you choose. Standing up a hosted gateway is typically a same-day task: create an account, add provider keys, point your application at the gateway's endpoint, and define a fallback rule or two. Building a self-hosted router with a library like LiteLLM usually takes a few days to a couple of weeks for a first production-ready version, including normalizing your existing calls, wiring in retries and timeouts, and adding logging. A fully custom router built from scratch — writing your own adapters for each provider, your own retry and load-balancing logic, and your own observability — realistically takes one to three weeks of focused engineering time for a basic version, and ongoing maintenance every time a provider changes its API.

Ongoing operational cost also includes the ongoing maintenance burden of provider APIs changing shape. Any provider can adjust rate limits, deprecate a model version, or change response fields with fairly short notice, and every router — hosted or self-hosted — needs someone watching provider changelogs and updating adapters or configuration accordingly. Hosted gateways absorb most of this burden on your behalf as part of what you're paying for; self-hosted setups put that maintenance on your own team.

Where teams get it wrong

The most common mistake is treating routing as a one-time setup rather than an operational system. Teams wire up a primary-plus-fallback configuration, verify it works once in testing, and never revisit it — then months later discover the fallback path has been silently broken (a stale API key, a renamed model) and would have failed exactly when it was needed, because nobody was watching whether the fallback path itself stayed healthy.

A second frequent error is downgrading silently on fallback. If your primary is a frontier model and your fallback is a much smaller, cheaper model, a user during an outage gets a materially worse answer with no indication anything changed. Better practice is either to fail over to a comparable-quality model from a different provider, or to surface a lightweight signal (even just in logs or a status field) so degraded responses are traceable rather than invisible.

Third, teams underinvest in observability until it's too late. Without per-provider latency, error-rate, and cost tracking, you cannot tell whether your router is actually improving reliability or just adding a hop that occasionally makes things worse. It's common to find, after adding real monitoring, that a "fallback" provider has a much higher latency than expected and is quietly degrading the user experience every time it's invoked, or that one provider is silently consuming a disproportionate share of the budget because a routing rule was left pointed at an expensive model by default.

Fourth, retry storms. A naive router that retries aggressively against a provider that's already rate-limiting you makes the problem worse, not better — each retry consumes more of your rate-limit budget and can trip additional throttling. Sound implementations use exponential backoff between retries and cap the total number of attempts (commonly two to three) before failing over to a different provider entirely, rather than hammering the same one.

Fifth, teams sometimes route purely on cost and ignore output-quality regressions. Cost-based routing that swaps to a cheaper model whenever budget pressure rises can quietly degrade the product for end users if nobody is tracking task-specific quality metrics alongside spend — a support-ticket classifier or a code-generation task can tolerate a cheaper model far worse than a casual chat response can, so blanket cost-based routing without task-aware guardrails is a common source of quality complaints that take weeks to trace back to a routing change.

Finally, vendor lock-in creeps back in through the back door. Teams adopt a router specifically to avoid single-provider dependency, then build application logic that assumes one provider's specific quirks (its function-calling format, its context window, its exact error codes), which quietly re-couples the app to that provider even though the router theoretically supports others. Keeping the application layer strictly agnostic to any single provider's response quirks is what actually delivers the portability a router is meant to provide.

Decision framework: when to choose what

Choosing between a hosted gateway, a self-hosted library, and a fully custom router comes down to three questions: how much engineering time you have, how much control over data and infrastructure you need, and how deep your observability requirements are.

If you're an individual developer or a small team that wants to test multiple models and needs basic automatic fallback with minimal setup, a hosted gateway with a single unified API key and transparent per-token pricing is the fastest path — you're trading a small markup for zero infrastructure burden. If you're a production team that needs granular control over fallback chains, budget alerts, and real-time observability into every provider's latency and error rate, a hosted gateway with stronger observability and configurable routing rules is usually worth the added subscription cost, since building equivalent tooling yourself takes real engineering time. If your team has strong Python or Node engineering resources and needs to keep data on your own infrastructure, or wants routing logic that's fully custom to your product (routing by user tier, content type, or a proprietary cost model), a self-hosted library gives you the normalization and provider adapters for free while leaving you in control of the actual routing rules and data path.

Regardless of which path you pick, the decision is rarely permanent. Many teams start with a hosted gateway to validate the product, then migrate to a self-hosted setup once request volume and data-residency requirements justify the engineering investment — because the normalized request/response shape a router enforces is exactly what makes that later migration a configuration change rather than an application rewrite.

Related questions

What's the difference between load balancing and fallback in LLM routing?

Load balancing actively splits normal traffic across multiple healthy providers (for cost or throughput reasons), while fallback only engages when a primary provider fails or times out. Most production routers use both together.

Do I need a router if I only use one LLM provider today?

Not immediately, but adding a normalization layer early — even with one provider configured — makes adding a second provider later a configuration change instead of an application rewrite.

Can routing reduce LLM costs without hurting quality?

Yes, if routing is task-aware: send routine, low-stakes tasks to cheaper models and reserve frontier models for tasks that need them, while tracking quality metrics per task type to catch regressions.

How is routing different from using an orchestration framework like LangChain?

Routing specifically decides which provider/model serves a request; orchestration frameworks handle broader application logic (chains, agents, retrieval) and often include routing as one feature among many.

What happens to conversation context when a router fails over to a different provider mid-conversation?

The router needs to resend the full conversation history to the new provider, since providers don't share session state — this is why normalizing message format up front matters for seamless failover.

FAQ

What is the simplest way to start routing requests across multiple LLM providers? Start with a hosted gateway that exposes a single unified API key and lets you specify a primary model and a fallback model. This requires no infrastructure and gets basic resilience in place within an afternoon.

How do I handle a provider being down or rate-limited? Configure automatic fallback to a secondary provider with a short timeout (roughly 8-20 seconds) and a capped retry count, using exponential backoff between attempts so you don't worsen an existing rate limit.

Can I track spending across multiple providers in one place? Yes — hosted gateways with observability dashboards and self-hosted libraries with logging integrations can both aggregate per-provider token usage and cost into a single view, which is essential once you're routing across more than one vendor.

Is it better to route by cost or by quality? Neither alone — route by task requirements first (does this task need frontier quality?), then optimize for cost within that quality band. Pure cost-based routing without quality guardrails is a common source of silent product regressions.

Do multi-provider routers support open-weight or self-hosted models alongside commercial APIs? Many do — self-hosted libraries in particular can route to locally-run open-weight models the same way they route to commercial provider APIs, as long as the local deployment exposes a compatible request/response interface.

How much engineering time does building a custom router take? A basic custom router covering a few providers, retries, and fallback typically takes one to three weeks of focused engineering time, plus ongoing maintenance whenever a provider changes its API.

Sources

• OpenRouter documentation
• Portkey AI Gateway documentation
• LiteLLM GitHub repository
• LangChain documentation
• OpenAI API documentation
• Anthropic API documentation
• Google AI for Developers documentation
• AWS: Implementing retry and backoff strategies

Related on PULSE

• [The 10 Best LLM Gateways in 2027](/knowledge/ai354)
• [The 10 Best LLM Routing and Load Balancing Tools in 2027](/knowledge/ai412)
• [The 10 Best Foundation Model API Providers in 2027](/knowledge/ai434)
• [How do you secure an LLM application's infrastructure?](/knowledge/ai363)
• [How do you evaluate LLM output quality at scale?](/knowledge/ai367)
• [The 10 Best GPU Cloud Providers for AI Training in 2027](/knowledge/ai340)