What is model serving and how is it different from a REST API?

What model serving actually is and why the distinction matters

The confusion is understandable, because for most teams the first production model *was* a REST API. Someone wrapped model.predict() in a Flask route, put it behind a load balancer, and called it deployed. That works. It works right up until the moment traffic triples, or the data scientist ships version 4 of the model, or the GPU that cost $2,000 a month sits at 8% utilization while requests queue behind each other one at a time.

A REST API is a protocol contract: resources addressed by URL, verbs like POST and GET, JSON in and JSON out, stateless requests, status codes that mean something. It says nothing about what happens after the request lands. It does not know that a PyTorch model takes 400 milliseconds to load from disk, that a GPU processes a batch of 32 images in barely more wall-clock time than a batch of one, or that you might want 10% of traffic hitting a challenger model while 90% stays on the incumbent.

Model serving is the set of concerns that live behind that door. Concretely, a serving layer owns at least six things a bare web framework does not:

Model lifecycle. Loading serialized weights into memory once at startup rather than per request, keeping them warm, and unloading them when a model goes idle. A ResNet-50 checkpoint is roughly 100 MB; a 7-billion-parameter LLM in 16-bit precision is about 14 GB. The difference between loading that once and loading it per request is the difference between a 30-millisecond response and a 30-second one.

Versioning and rollout. Serving frameworks treat a model as a versioned artifact with a name and a number. TensorFlow Serving watches a directory and hot-swaps when a new numbered subdirectory appears. Triton reads a model repository with per-model config files. KServe expresses the whole thing as a Kubernetes InferenceService custom resource. In all three cases you can roll forward and roll back without redeploying the surrounding application.

Batching. This is the single biggest reason serving frameworks exist. GPUs are throughput devices; they are catastrophically underused by one-request-at-a-time processing. Dynamic batching collects incoming requests inside a short window — typically 1 to 100 milliseconds, configurable — and submits them as one tensor. You trade a small, bounded latency penalty for a large throughput gain.

Hardware scheduling. Which model gets which GPU, how many concurrent model instances run per device, whether to use TensorRT or ONNX Runtime or plain eager-mode PyTorch as the execution backend, whether to enable mixed precision.

Protocol multiplexing. Most serving frameworks expose the *same* loaded model over both REST and gRPC. Triton, TorchServe, TensorFlow Serving, Seldon Core, and KServe all do this. That fact alone should settle the question: if REST and gRPC are two interchangeable front doors onto one serving engine, then REST cannot be the serving layer.

Observability. Per-model latency percentiles, queue depth, batch size distribution, GPU utilization, and success/failure counts, usually exposed on a Prometheus endpoint.

The adjacent point worth making: this same split shows up everywhere in infrastructure, not just ML. A database has a wire protocol and a storage engine. A search cluster has a REST interface and an inverted-index engine. Nobody says "Elasticsearch is a REST API," even though you talk to it almost exclusively over REST. Model serving deserves the same distinction.

How a serving request actually flows end to end

Walk a single prediction through a production serving stack and the layers separate cleanly.

Step 1 — Client call. A caller POSTs JSON to something like /v2/models/fraud-scorer/versions/3/infer. This is the REST part, and it is genuinely thin: parse the body, validate the shape, return a status code. In FastAPI this is a Pydantic model and a decorator. In Triton it is a KServe-standard v2 inference protocol endpoint.

Step 2 — Deserialization and validation. JSON is a poor container for tensors. A 224×224×3 float32 image encoded as a JSON array of numbers is roughly 10× the size of the raw binary. This is exactly why high-throughput serving stacks prefer gRPC with protobuf, or REST with a binary tensor extension. If your payloads are images, audio, or embeddings, the serialization cost can exceed the inference cost.

Step 3 — Queue and batch. The request enters a per-model queue. The scheduler either fires immediately if a batch is full, or waits up to the configured maximum queue delay. Set that window to 5 ms and you add at most 5 ms of latency; set it to 100 ms and you get fatter batches but a visibly slower p99.

Step 4 — Execution. The batched tensor is handed to the backend — TensorRT, ONNX Runtime, LibTorch, XLA, or a custom Python backend. The model is already resident in GPU memory; nothing loads here.

Step 5 — Unbatch and respond. Outputs are split back to their originating requests, serialized, and returned with the appropriate status code.

Step 6 — Emit telemetry. Latency split into queue time versus compute time, batch size, and success/failure, all pushed to metrics.

Notice what steps 1, 2, and 5 have in common: they are the REST API. Steps 3, 4, and 6, plus the version-watching path, are the serving layer. You can delete REST entirely, put gRPC in its place, and every interesting part of the diagram is unchanged.

The upstream and downstream halves matter too. Upstream, a feature store or preprocessing service often sits between the caller and the model — the raw request carries a user ID, and something has to turn that into a 200-dimensional feature vector before inference. Downstream, predictions usually get logged to a warehouse for drift monitoring and eventual retraining. A serving framework that gives you request/response logging hooks for free saves you building that pipeline by hand.

Costs, latency budgets, and the ranges you should plan around

Numbers here vary enormously by model and hardware, so treat these as shapes rather than guarantees, and benchmark your own workload before committing.

Software cost is essentially zero. Triton, TorchServe, TensorFlow Serving, BentoML, Ray Serve, KServe, Seldon Core, MLflow, ONNX Runtime, and FastAPI are all open source under permissive licenses (mostly Apache 2.0 or BSD). The entire cost conversation is compute, engineering time, and optionally a managed control plane.

Compute dominates. A single cloud GPU instance suitable for mid-size inference typically runs in the low hundreds of dollars per month if left on continuously; larger multi-GPU instances run into the thousands. The relevant question is not the hourly rate but the utilization. If dynamic batching takes a GPU from 10% utilization to 60%, you have effectively cut your per-prediction cost by roughly a factor of six without changing the bill. That is the actual ROI argument for a serving framework, and it is why "just use Flask" gets expensive at scale even though Flask is free.

Latency budget, decomposed. For a well-tuned small-model endpoint, expect network and TLS overhead in the single-digit milliseconds, JSON serialization in the low single digits for small payloads and much more for tensors, queue wait bounded by whatever you configured, and compute in the low tens of milliseconds for a small CNN or classical model. LLM generation is a completely different regime — time-to-first-token in the hundreds of milliseconds and then a per-token stream — which is why LLM serving uses continuous batching and paged attention rather than the fixed-window dynamic batching that works for vision models.

Engineering time. Wrapping a scikit-learn model in FastAPI is an afternoon. Getting a Triton model repository configured with correct input/output shapes, dynamic batching parameters, and instance groups is typically a few days for someone who has not done it before, and the config files are where people lose time. A Kubernetes-native platform like KServe or Seldon Core adds the cost of the cluster itself — if you do not already run Kubernetes, adopting it *for model serving alone* is rarely the right trade.

Scale-to-zero economics. Knative-backed serving (which KServe uses) can drop replicas to zero when idle, which is transformative for internal tools and long-tail models that see a few requests an hour. The catch is cold start: reloading a multi-gigabyte model can take tens of seconds. Scale-to-zero is excellent for bursty internal traffic and usually wrong for user-facing latency-sensitive endpoints.

A rough cost ladder. One model, low traffic, no GPU: FastAPI plus ONNX Runtime on a small CPU instance, near-trivial cost. A handful of models with real traffic on one GPU: a single Triton or TorchServe container with concurrent model execution, one instance instead of five. Dozens of models across teams: a Kubernetes platform with per-model autoscaling, where the platform overhead finally pays for itself.

Where teams get this wrong

Mistake one: treating the REST wrapper as the whole deployment. The Flask endpoint ships, everyone celebrates, and six months later there is no way to answer "which model version produced this prediction?" Version identity should be in the response, in the logs, and in the URL path from day one. Retrofitting it after an incident is painful.

Mistake two: loading the model inside the request handler. It sounds absurd written down, but it happens constantly with lazy-loading patterns and with serverless functions that get a cold container per invocation. Load at startup, hold the reference in module scope or an app-state object, and add a readiness probe that only passes once the model is resident.

Mistake three: running one worker per CPU core with a large model. Gunicorn with eight workers and a 4 GB model is 32 GB of RAM for one endpoint, and every worker duplicates the weights. Serving frameworks solve this with a shared model instance and internal concurrency; naive WSGI scaling does not.

Mistake four: setting the batching window without measuring. A max queue delay of 100 ms feels harmless until you look at a p99 that is now 100 ms worse for every request that arrives in a quiet period. Start small — 2 to 10 ms — and increase only while watching throughput gain against tail latency.

Mistake five: shipping JSON tensors at volume. If you are sending image arrays as JSON numbers, you are burning CPU on serialization on both ends and inflating payloads several-fold. Send base64-encoded compressed images and decode server-side, or switch to gRPC with binary tensors.

Mistake six: no shadow or canary path. Because "deploying a model" felt like "deploying an API," the rollout inherits the API's all-or-nothing deploy. Serving platforms give you traffic splitting and shadow deployments — send a copy of live traffic to the challenger, compare outputs offline, promote only when the comparison is clean. Not using that capability is leaving the main benefit on the table.

Mistake seven: no drift monitoring because "the API returns 200." HTTP status tells you the service is up. It tells you nothing about whether input distributions have shifted or the model has quietly gotten worse. Log inputs and predictions (with appropriate privacy controls), and compare distributions against the training set on a schedule.

Mistake eight: adopting Kubernetes-native serving before having Kubernetes. KServe and Seldon Core are excellent when a cluster and a platform team already exist. When they do not, the operator, the ingress, the service mesh, and the CRDs become the project.

Mistake nine: assuming a general-purpose serving framework handles LLMs well. Autoregressive generation needs continuous batching and KV-cache paging, not fixed-window batching over a static graph. Use a runtime built for it rather than forcing a vision-shaped serving pattern onto text generation.

A decision framework for picking the serving layer

Pick along three axes: how many frameworks you must support, whether you already run Kubernetes, and whether the bottleneck is throughput or developer velocity.

Multi-framework and GPU-bound → Triton. It serves PyTorch, TensorFlow, ONNX, TensorRT, and custom Python or C++ backends from one process, with dynamic batching, concurrent model execution, model ensembles that chain preprocessing and inference server-side, and both REST and gRPC front ends. The cost is configuration complexity.

Python-first and velocity-bound → BentoML or Ray Serve. BentoML generates the REST API and OpenAPI docs from decorated Python, handles adaptive batching, and containerizes with one command — the shortest honest path from notebook to production. Ray Serve is the better fit when inference is one stage of a multi-step Python pipeline or when you already run Ray for distributed work.

Single-framework shops → the native server. TorchServe for PyTorch, TensorFlow Serving for TensorFlow. Both give versioning, batching, management endpoints, and dual REST/gRPC with far less setup than a general-purpose server. There is no prize for generality you do not need.

Kubernetes-native with advanced rollout needs → KServe or Seldon Core. Both express a deployment as a custom resource, integrate with the cluster's autoscaling and ingress, and support canary traffic splitting and explainability integrations. KServe leans serverless via Knative including scale-to-zero; Seldon Core leans toward inference graphs and richer traffic-routing strategies.

Already on MLflow → MLflow serving. If experiment tracking and the model registry already live in MLflow, serving a registered model directly is the least-friction option for internal and batch-adjacent workloads. It is deliberately simple and does not aim at GPU-optimized high-throughput serving.

Low traffic, tight budget, one model → FastAPI plus ONNX Runtime. Export to ONNX, run it on CPU, get automatic OpenAPI docs and input validation. Graph optimizations and quantization in ONNX Runtime meaningfully reduce CPU latency. This is a genuinely good answer for a large share of real workloads — most models are not LLMs and do not need a GPU.

The migration path matters as much as the starting point. The pattern that works is: start with the simplest thing that serves correct predictions, instrument it properly, and let measured pain — not anticipated pain — drive the move. Teams that jump straight to a Kubernetes serving platform for their first model usually spend more time on the platform than the model.

What changes when serving meets the rest of the stack

Serving does not live alone. A few adjacent effects are worth planning for.

Feature freshness. If features are computed at training time in a batch pipeline and at serving time in a request handler, the two implementations drift. Training/serving skew is among the most common causes of a model that scores well offline and disappoints in production. Either compute features once and read them from a shared store, or share the transformation code between both paths.

Preprocessing placement. Image decoding, tokenization, and normalization can run client-side, in the serving process, or as an ensemble step inside the serving framework. Pushing them server-side keeps clients thin and guarantees consistency; keeping them client-side reduces payload size. Triton's ensemble models and Seldon's inference graphs exist precisely to make server-side preprocessing a first-class, versioned artifact rather than ad-hoc code in a route handler.

Batch versus online. Not every prediction needs a synchronous REST call. If consumers read predictions from a table the next morning, a scheduled batch job is cheaper, simpler, and easier to backfill. The serving question only becomes interesting when a human or a system is waiting on the answer.

Testing. Model endpoints need contract tests (does the schema hold?), golden-output tests (does a fixed input still produce the expected prediction for a given version?), and load tests (what happens at 5× traffic?). The golden-output test is the one teams skip and the one that catches a silently wrong version promotion.

Security and multi-tenancy. Inference endpoints leak information — through timing, through confidence scores, and through logged inputs that may contain personal data. Rate-limit per caller, be deliberate about what the response exposes, and scrub logs. Multi-model servers add a tenancy question: one team's runaway batch can starve another team's latency-sensitive model on a shared GPU, so per-model resource limits are not optional.

Organizational ownership. The most durable pattern is a small platform team owning the serving layer and the deployment contract, while data science teams own model artifacts and configs. When every team runs its own bespoke Flask service, nobody owns latency regressions and the same batching bug gets fixed five times.

Related questions

Can a REST API be model serving if it does everything a serving framework does?

Yes — the distinction is functional, not technological. If your FastAPI service loads models once, batches requests, tracks versions, and exports metrics, you have built a serving layer that happens to speak REST. You have just built it yourself instead of adopting one.

Why do serving frameworks offer gRPC alongside REST?

gRPC uses HTTP/2 with binary protobuf payloads, multiplexed connections, and native streaming. For tensor payloads that means smaller messages and less CPU spent on serialization. REST stays for browser clients, easy debugging with curl, and broad tooling support.

Does model serving apply to non-ML models?

The pattern generalizes. Any expensive-to-initialize, stateful compute artifact — a rules engine, an optimization solver, a large in-memory index — benefits from the same load-once, batch, version, and observe pattern. ML just made the pattern common enough to name.

How does serving differ for large language models specifically?

Autoregressive generation produces one token at a time, so fixed-window batching fits poorly. LLM runtimes use continuous batching, where finished sequences leave the batch and new ones join mid-flight, plus paged KV-cache management to avoid wasting GPU memory on padding.

Is a managed endpoint better than self-hosting the serving layer?

Managed endpoints trade cost and control for time. They are usually right for the first few models and for teams without platform engineers. Self-hosting wins once utilization is high enough that the managed premium exceeds the salary cost of operating it yourself.

FAQ

What is the main difference between model serving and a REST API?

Model serving is the complete runtime that loads models, batches requests, manages versions, schedules hardware, and emits inference telemetry. A REST API is one transport interface exposing that runtime over HTTP. Most serving frameworks expose the same model over REST *and* gRPC simultaneously, which makes the layering explicit: the protocol is interchangeable, the serving engine is not.

Do I need a serving framework if I already have a working REST endpoint?

Not necessarily. If you serve one small model at modest traffic and have no versioning pain, a well-built FastAPI service is a legitimate production answer. Adopt a framework when a specific problem appears: GPU sitting idle, no safe rollback, memory duplicated across workers, or no per-model latency visibility. Migrate on evidence, not anticipation.

Can Flask or FastAPI alone handle production model serving?

They can, with work. You would implement request batching, model registry integration, warm loading, health and readiness probes, and metrics yourself — all things a serving framework ships. FastAPI is the better base of the two because it is ASGI-native, gives automatic OpenAPI docs, and does typed request validation through Pydantic without extra code.

What is dynamic batching and when does it actually help?

Dynamic batching holds incoming requests for a short configurable window and submits them to the accelerator as one tensor. It helps most when compute is GPU-bound and per-request work is small relative to kernel launch overhead — vision models, embedding generation, ranking. It helps least on CPU-bound classical models, where batching adds latency without meaningful throughput gain.

How should model versions be handled in production?

Treat models as immutable numbered artifacts in a registry, never overwrite in place, and make the version explicit in the request path or response body. Keep the previous version loaded and serving so rollback is a routing change rather than a redeploy. Log the version alongside every prediction so incidents are traceable to a specific artifact.

How do I know whether a serving framework is actually paying for itself?

Measure three things before and after: GPU or CPU utilization under production traffic, p50 and p99 latency, and predictions served per dollar of compute per day. If utilization rose substantially and p99 held within budget, the framework earned its complexity. If all three are unchanged, you added operational surface for nothing — a valid reason to roll back to the simpler service.

Sources

• NVIDIA Triton Inference Server documentation
• TorchServe documentation
• TensorFlow Serving guide
• BentoML documentation
• Ray Serve documentation
• KServe documentation
• Seldon Core repository
• MLflow deployment documentation
• ONNX Runtime documentation
• FastAPI documentation

Related on PULSE

• [The 10 Best Model Serving Frameworks in 2027](/knowledge/ai372)
• [How do you A/B test different LLMs in production?](/knowledge/ai391)
• [How do you reduce GPU costs when serving large language models?](/knowledge/ai343)
• [The 10 Best Foundation Model API Providers in 2027](/knowledge/ai434)
• [The 10 Best AI Tools for REST API Development in 2027](/knowledge/ai0223)
• [The 10 Best AI Tools for API Testing in 2027](/knowledge/ai0078)