What is the difference between batch and real-time inference infrastructure?

What it is and why it matters

The distinction between batch and real-time inference is less about the model and more about who is waiting. In batch inference, nobody is waiting: a scheduler kicks off a job, the job reads a bounded set of rows from a warehouse or object store, runs them through a model, and writes predictions back to a table. In real-time inference, a user or an upstream service is blocked on the answer, and every millisecond of latency is a millisecond of someone's session.

That single difference cascades through every layer of the infrastructure. Batch systems can afford cold starts, because a thirty-second container spin-up is noise inside a forty-minute job. Real-time systems cannot, which is why serverless real-time inference on functions like AWS Lambda is awkward — a one-to-three second cold start is catastrophic for a p99 latency target of 100 milliseconds. Batch systems can use preemptible or spot instances, because a lost worker just re-runs its partition. Real-time systems generally cannot, because losing a node mid-request means dropped traffic and a paged on-call engineer.

The same asymmetry shows up in how compute is provisioned. A batch job is allowed to saturate its hardware — you want GPU utilization at 90%+ during the run, and 0% the rest of the day, because you only pay for the window. A real-time endpoint that runs at 90% utilization has no headroom for a traffic spike, so most teams provision real-time capacity at 30–50% steady-state utilization and accept that they are paying for idle silicon as insurance against tail latency. That insurance premium is the single biggest cost driver in real-time serving, and it is why the same model can cost ten to fifty times more per prediction when served online than when scored offline.

Data freshness is the other axis. Batch predictions are as fresh as the last run: a nightly lead-scoring job means the score a rep sees at 9 a.m. reflects yesterday's behavior. For many revenue workflows that is completely acceptable — a territory assignment or a propensity tier does not change meaningfully in six hours. But for fraud scoring, dynamic pricing, in-session recommendations, or routing an inbound demo request while the form is still on screen, a stale prediction is a wrong prediction. The decision of batch versus real-time is therefore a product decision disguised as an infrastructure decision: how quickly does the world change relative to how quickly your score is consumed?

There is a third mode worth naming, because teams routinely conflate it with real-time: streaming or near-real-time inference. Here, a consumer reads from Kafka, Kinesis, or Pub/Sub, scores micro-batches every few seconds, and writes results to a low-latency store. Latency lands in the seconds-to-a-minute range, cost sits between the two extremes, and the operational model resembles batch more than online serving. A surprising number of "we need real-time" requirements are satisfied perfectly by streaming at a fraction of the cost, and asking "what is the actual staleness tolerance in seconds?" is the fastest way to find out.

The step-by-step process

Building either path follows a recognizable sequence, but the sequences diverge sharply after step two.

Batch inference pipeline. First, define the population — the set of entities to score. This is a SQL query against the warehouse: all accounts active in the last 90 days, all SKUs in the current catalog, all open opportunities. Second, assemble features, usually by joining behavioral aggregates computed in the same warehouse. Third, load the serialized model artifact into a distributed compute engine — Spark with MLeap, Ray, a Vertex AI batch prediction job, a SageMaker transform job, or a plain container running pandas if the volume is modest. Fourth, score in partitions, because a single worker cannot hold ten million rows in memory. Fifth, write results to a durable sink: a warehouse table, a Parquet file in object storage, or a reverse-ETL push into the CRM. Sixth, validate — row counts, null rates, and score distribution compared against the prior run. Seventh, publish the run's metadata so downstream consumers know which model version produced which scores.

Real-time inference pipeline. Steps one and two collapse into a request payload plus a feature lookup. The request carries whatever the caller knows; everything else must be fetched in single-digit milliseconds from a low-latency store — Redis, DynamoDB, or a managed online feature store. This feature-fetch step is where most real-time latency budgets are quietly consumed, and where the infamous training-serving skew originates: the batch pipeline computed a 30-day rolling average with a warehouse window function, and the online path approximates it with a counter that resets differently. Third, the request hits a model server — Triton, TorchServe, TensorFlow Serving, Ray Serve, BentoML, Seldon Core, or a managed endpoint on SageMaker, Vertex AI, or Azure ML. Fourth, the server applies dynamic batching, holding requests for a few milliseconds to group them into a GPU-efficient batch. Fifth, post-processing and response. Sixth, the prediction is logged asynchronously for monitoring — never synchronously, because logging in the request path adds latency you cannot afford.

The step that teams most often skip is validation on the batch side and async logging on the real-time side. A batch job that silently scores 40% fewer rows than yesterday because an upstream join broke will happily overwrite good scores with a partial set, and nobody notices until a rep asks why their whole territory went cold. On the real-time side, an endpoint with no prediction logging is a black box — you cannot detect drift, you cannot reconstruct why a decision was made, and you cannot build the training set for the next model version.

Dynamic batching deserves its own note because it is the mechanism that lets one server straddle both worlds. The server waits a configurable window — often 1 to 10 milliseconds — accumulating requests up to a maximum batch size, then runs them as one forward pass. On a GPU this can improve throughput by three to ten times versus scoring one request at a time, at the cost of adding the wait window to every request's latency. Tuning it is a direct trade: a larger max batch and longer timeout raises throughput and raises p99 latency. Triton's model analyzer and perf analyzer tools exist specifically to search that space empirically rather than by guesswork.

Costs, timelines, and typical ranges

Cost structure is the cleanest way to explain the difference to a finance partner. Batch inference cost is roughly *(instance hourly rate) × (job duration) × (runs per period)*, and job duration shrinks as you add workers, so the cost is close to fixed per unit of work regardless of how you parallelize. Real-time cost is *(instance hourly rate) × 24 × (number of replicas)*, and it is almost entirely decoupled from how many predictions you actually serve. An endpoint serving 100 requests per day and one serving 100,000 may cost exactly the same if both need two replicas for redundancy.

Public cloud pricing gives usable anchors. Managed real-time endpoints on the major clouds land roughly in the $0.10–$0.50 per vCPU-hour range for CPU instances, and GPU-backed endpoints commonly run from well under a dollar per hour for a small T4 or L4 instance up into the tens of dollars per hour for multi-A100 or H100 machines. Because a production endpoint typically runs at least two replicas across availability zones, the floor for a *single* always-on GPU-backed real-time model is meaningful monthly spend even at zero traffic. Batch jobs on the same hardware family cost the same per hour but run for a bounded window, and spot or preemptible pricing frequently cuts that by 60–90%. Always check current published pricing pages before quoting numbers to a stakeholder — rates move, and instance families are added and retired continuously.

Timelines follow a similar asymmetry. A competent data team can stand up a first batch scoring pipeline in days: the warehouse already exists, the orchestrator (Airflow, Dagster, Prefect, Step Functions) already exists, and the job is a container plus a DAG. A production-grade real-time endpoint takes considerably longer — commonly several weeks to a couple of months — because the work is not the model server, it is everything around it: an online feature store, autoscaling policies, load testing, canary deployment, latency alerting, circuit breakers, and a rollback path. Teams consistently underestimate this by a factor of three, budgeting for the serving container and discovering the feature-fetch layer is the actual project.

Latency ranges worth internalizing: an optimized small model on a GPU server can serve in single-digit milliseconds of pure compute; add feature lookup, network hops, serialization, and load-balancer overhead and end-to-end p50 for a well-built endpoint typically lands in the tens of milliseconds, with p99 several times higher. Batch throughput on a modern data-center GPU is measured in tens of thousands of inferences per second for compact models, which is why a job scoring ten million rows can finish in minutes rather than hours. Large language models invert these numbers entirely — token-by-token generation means latency scales with output length, and batch LLM jobs are usually priced per token rather than per instance-hour.

The economically interesting middle ground is multi-model endpoints, where many low-traffic models share one instance and are loaded on demand. If you have fifty customer-specific models each serving a handful of requests per minute, fifty dedicated endpoints is indefensible; one shared endpoint with lazy loading turns fifty idle bills into one, at the cost of a cold-load penalty on the first request for an evicted model. This pattern is how teams make per-tenant modeling economically viable at all, and it is available in some form on most managed platforms.

Where teams get it wrong

Building real-time when the consumer reads batch. The most common and most expensive mistake. A team builds a low-latency endpoint, wires it into a CRM field, and the field is read by a rep who opens the record twice a day. The score could have been computed nightly for a rounding error of the cost. Before committing to real-time, trace the consumption path: who reads the prediction, how often, and does their decision actually change if the number is six hours old? For a large share of revenue-facing use cases — lead scoring, churn risk tiers, expansion propensity, territory routing — the honest answer is no.

Training-serving skew from two feature paths. When batch features are computed in SQL and online features are computed in application code, they drift. The batch job's "sessions in last 7 days" uses a warehouse window; the online path uses a Redis counter with different reset semantics. The model degrades silently because it was trained on one distribution and serves against another. The structural fix is a feature store that defines each feature once and materializes it to both an offline and an online store, or, failing that, computing features in one place and shipping them to the other rather than reimplementing.

Load-testing with idle latency. A model that returns in 5 milliseconds on an empty server will not return in 5 milliseconds at 100 concurrent requests — queueing, memory bandwidth contention, and GPU scheduling push p99 far higher. Measure under realistic concurrency with realistic payload sizes before publishing an SLA. Tools like Triton's perf analyzer, or a generic load generator like k6 or Locust, will surface this in an afternoon and save a quarter of firefighting.

No versioning on batch outputs. Overwriting a scores table in place means you cannot answer "what score did this account have when the rep called it?" Append with a run timestamp and model version, and let consumers read the latest view. The storage cost is trivial; the auditability is not, especially when a prediction influences pricing, credit, or anything a customer might later dispute.

Ignoring the batch job's failure blast radius. Real-time failures are loud — error rates spike, alerts fire. Batch failures are quiet. A job that fails at 2 a.m. and is not alerted on leaves stale scores in place, and stale scores look exactly like fresh scores to every downstream consumer. Every batch pipeline needs a freshness check on the *consuming* side, not just a success check on the producing side: if the max score timestamp is older than the expected interval, alert.

Treating GPU as the default. GPUs are essential for large neural networks and transformer models. For gradient-boosted trees, logistic regression, and small networks — which still cover a large fraction of revenue and operations use cases — CPU inference is faster end-to-end once you account for data transfer overhead, and dramatically cheaper. Benchmark on CPU first; graduate to GPU when the numbers demand it.

Underestimating the orchestration surface. Batch inference is not one job; it is a dependency graph — features must be fresh before scoring, scoring must complete before reverse-ETL, reverse-ETL must complete before the morning report. When any link runs late, everything downstream is stale. Model the dependencies explicitly in the orchestrator with data-aware sensors rather than hoping a fixed cron offset holds forever.

Decision framework: when to choose what

Start with staleness tolerance measured in time units, not adjectives. If the acceptable age of a prediction is measured in hours, choose batch. If it is measured in seconds to minutes, choose streaming. If it is measured in milliseconds, choose real-time. This single question eliminates most of the debate, and it is answerable by the person who consumes the prediction rather than the person who builds it.

Then layer on volume and cardinality. If the population you would score in batch is small and bounded — say, 50,000 accounts — precomputing every score nightly costs almost nothing and gives you real-time-feeling reads from a plain database lookup. If the input space is effectively unbounded — every possible search query, every session context, every combination of cart contents — precomputation is impossible and real-time is forced regardless of your latency preference. This "can I enumerate the inputs?" test is often more decisive than the latency test.

A practical hybrid deserves emphasis because it is underused: precompute the expensive part, serve the cheap part live. Run the heavy feature engineering and the bulk of the model offline, store an embedding or a base score per entity, and at request time combine that stored artifact with the handful of live signals that actually changed. Recommendation systems have done this for years — candidate generation offline, ranking online — and the pattern transfers cleanly to revenue use cases: compute an account's base propensity nightly, adjust it live for what the visitor is doing in this session. You get most of the freshness for a fraction of the online compute.

On tooling, the honest guidance is to choose by constraint rather than by feature matrix. If your team already runs Spark for data engineering, batch inference inside existing Spark pipelines adds near-zero infrastructure. If you are Kubernetes-native and need multi-model pipelines, a Kubernetes-native serving platform fits the operational model you already have. If you need one server to handle mixed frameworks and both modes on GPU, a general-purpose inference server with dynamic batching and concurrent model execution is the strongest single choice. If you want managed and you are already committed to a cloud, that cloud's managed endpoints will be the fastest path to production and the slowest path to portability. There is no universally correct answer; there is only the answer that matches your existing operational competence.

Finally, revisit the decision on a schedule. Traffic patterns change, models get larger, and a workload that justified a dedicated GPU endpoint at launch may be better served six months later by a multi-model endpoint or a quantized CPU deployment. Put a quarterly review of inference spend and latency percentiles on the calendar the same way you would review any other infrastructure line item.

Related questions

Can one platform serve both batch and real-time inference?

Yes. General-purpose inference servers and the major managed ML platforms all expose both a low-latency endpoint mode and an offline transform mode. The benefit is one model artifact and one deployment process; the caveat is that the optimal hardware for each mode still differs.

What is dynamic batching and when does it hurt?

Dynamic batching holds incoming requests for a short window to group them into one GPU pass, raising throughput substantially. It hurts when traffic is sparse — a request arriving alone still pays the full wait window, inflating latency for no throughput gain. Shorten the window under low load.

How do I keep batch and real-time features consistent?

Define each feature once and materialize it to both an offline store for training and an online store for serving. If a feature store is not available, compute in one place and replicate to the other. Never reimplement the same feature logic twice in different languages.

Is streaming inference a separate category?

Practically, yes. It reads from a message bus, scores micro-batches every few seconds, and writes to a low-latency store. Latency lands between batch and real-time, cost is closer to batch, and it satisfies many requirements that were mislabeled as needing true real-time.

Does the batch versus real-time choice change for LLMs?

The framing holds but the economics shift. LLM latency scales with generated tokens rather than being roughly constant, batch LLM work is typically billed per token, and offline batch APIs from model providers often price at a substantial discount versus synchronous calls for the same work.

FAQ

What is the single biggest difference between batch and real-time inference infrastructure?

Whether a caller is blocked waiting for the result. That determines everything downstream: batch can queue, retry, use spot instances, and tolerate cold starts, while real-time must hold idle capacity, avoid cold starts, and guarantee a tail-latency percentile. The models can be identical; the surrounding infrastructure is not.

Why is real-time inference so much more expensive per prediction?

Because you pay for capacity, not usage. Real-time endpoints run continuously with headroom for spikes, typically at 30–50% steady-state utilization, and usually with redundant replicas. Batch jobs run only during the scoring window at near-full utilization and can often use discounted preemptible capacity, so the same work costs far less.

Do I need GPUs for real-time inference?

Not always. Gradient-boosted trees, linear models, and small neural networks frequently serve faster and far cheaper on CPU once data-transfer overhead is counted. GPUs become necessary for large neural networks and transformer models where compute dominates. Benchmark on CPU first and move to GPU only when measurements justify it.

How do I decide the staleness tolerance for a revenue use case?

Ask the person consuming the score how often they act on it and what changes their decision. A rep reviewing a territory weekly does not need sub-second freshness. A pricing engine responding to an in-session visitor does. Write the tolerance down in seconds — that number, not a preference, drives the architecture.

What breaks most often in batch inference pipelines?

Silent staleness. The job fails or scores a partial population, and downstream consumers keep reading yesterday's numbers as if they were current. Guard against it with a freshness check on the consuming side, row-count and score-distribution validation on each run, and versioned appends rather than in-place overwrites.

Can I start with batch and migrate to real-time later?

Usually yes, and it is often the right sequence. Batch proves the model has value at low cost and low operational risk. Migrating means adding an online feature store, a serving layer, and load testing — real work, but work you undertake with evidence that the model matters. Design features to be computable both ways from the start to keep the door open.

Sources

• NVIDIA Triton Inference Server documentation
• Amazon SageMaker batch transform
• Amazon SageMaker real-time inference
• Google Vertex AI predictions overview
• Azure Machine Learning endpoints
• Ray Serve documentation
• TensorFlow Serving performance guide
• TorchServe documentation
• BentoML adaptive batching guide
• Feast feature store concepts

Related on PULSE

• [What is the difference between vLLM, TGI, and Triton for LLM inference?](/knowledge/ai345)
• [How do you choose between cloud GPUs and on-prem for AI workloads?](/knowledge/ai375)
• [What infrastructure do you need to run AI agents in production?](/knowledge/ai373)
• [The 10 Best Real-Time ML Feature Platforms in 2027](/knowledge/ai382)
• [How do you secure an LLM application's infrastructure?](/knowledge/ai363)