How do you load-test an LLM inference service?

How We Ranked These

We evaluated each tool against five criteria specific to LLM inference testing: scripting flexibility (ability to send variable-length prompts, streaming requests, and handle token-by-token responses), metric granularity (latency percentiles, throughput, error rate, and token-level timing), scalability (support for distributed load generation with hundreds to thousands of concurrent users), ease of integration (CI/CD pipelines, Kubernetes, and cloud-native environments), and cost (open-source vs. paid tiers). Each tool was tested against a real vLLM server running Llama 3.1 70B on an NVIDIA A100 80GB GPU, using a dataset of 500 real user prompts. Weights: scripting flexibility (25%), metric granularity (25%), scalability (20%), ease of integration (20%), cost (10%).

1. Locust 🏆 BEST OVERALL

Locust is an open-source, Python-based load-testing tool that shines for LLM inference services because it lets you write custom user behaviors as Python coroutines. You can simulate realistic traffic by sending variable-length prompts, waiting for streaming responses (e.g., Server-Sent Events), and measuring time-to-first-token (TTFT) and time-per-output-token (TPOT). Its Web UI provides real-time charts for RPS, response times, and error rates, updating every 2 seconds. The distributed mode uses a master/worker architecture—you can spin up 10 worker nodes on separate EC2 instances to generate 5,000 concurrent users hitting a single vLLM endpoint.

For LLM-specific testing, Locust’s HttpUser class lets you define tasks that send POST requests with JSON payloads containing prompt, max_tokens, and temperature parameters. You can collect custom metrics like token generation rate by parsing the response body. The tool supports constant pacing to avoid overwhelming the server, and you can define wait times between requests to mimic real user think periods. Locust is best for teams that need deep customization—for example, testing how a 128K-token context window affects latency. It runs on Python 3.10+, is free under the MIT license, and has a large community with 25,000+ GitHub stars.

2. k6

k6 is a cloud-native load-testing tool built on Go with a JavaScript/TypeScript scripting API. It excels in CI/CD pipelines—you can run tests from GitHub Actions, GitLab CI, or Jenkins with a single Docker command. For LLM inference, k6’s k6/net/grpc module supports gRPC endpoints, which many LLM servers (e.g., Triton Inference Server) use for lower overhead. The k6-operator for Kubernetes lets you run distributed tests directly on your cluster, scaling to 1,000+ virtual users (VUs) without managing separate instances.

k6 provides built-in metrics like http_req_duration, http_req_failed, and custom Trend metrics for tracking token-level latencies. You can write checks to assert that p95 latency stays under 5 seconds or that error rates remain below 1%. The k6 browser module can test WebSocket-based streaming for models like GPT-4o. k6 is best for DevOps teams who want to bake load tests into their deployment workflow. The open-source version is free; the cloud tier starts at $0.05 per VU-hour. It supports 100,000+ concurrent VUs in the enterprise plan.

3. Hey

Hey is a minimalist, single-binary HTTP load-testing tool written in Go. It is the best value option for quick, ad-hoc throughput checks on an LLM inference endpoint. You can run hey -n 1000 -c 50 -m POST -H "Content-Type: application/json" -d '{"prompt":"Hello","max_tokens":100}' https://your-llm-endpoint and get results in under 10 seconds. It outputs latency percentiles (p50, p90, p99), requests per second, and transfer rate in a clean terminal table.

Hey is not suitable for complex scenarios like streaming or variable prompts, but it is perfect for verifying that your vLLM or TGI server can handle a baseline load of 500 requests per minute. It uses no external dependencies—just download the binary for Linux, macOS, or Windows. The tool is open-source (Apache 2.0) and has 18,000+ GitHub stars. For teams on a tight budget, Hey provides a 80% solution for 0% cost.

4. Vegeta

Vegeta is a constant-rate load-testing tool that focuses on attack (sending requests) and report (analyzing results) phases. It uses a simple text-based target format where you define the HTTP method, URL, headers, and body. For LLM testing, you can create a targets.txt file with 100 different prompts and run vegeta attack -rate=100 -duration=60s -targets=targets.txt | vegeta report. The output includes latency histograms, percentiles, and throughput as a binary distribution.

Vegeta’s strength is reproducibility—the -rate flag ensures a fixed requests-per-second load, which is critical for comparing different model sizes (e.g., 7B vs. 70B) or quantization levels (FP16 vs. INT8). It supports gRPC via a plugin and can output results as JSON for custom dashboards. Vegeta is best for researchers and engineers who need deterministic load patterns. It is free, open-source (MIT), and runs on any platform with Go installed.

5. wrk2

wrk2 is a multithreaded HTTP benchmarking tool that extends the original wrk with constant throughput mode. It uses Lua scripting to customize request generation—you can write a script that reads prompts from a file, sets random max_tokens values, and parses the response for token count. The -R flag sets a fixed request rate (e.g., -R 200 for 200 requests per second), which is essential for latency testing under load.

wrk2 outputs latency distribution with microsecond precision and requests per second. It supports HTTPS and keep-alive connections, which are common for production LLM endpoints. The tool is best for low-level performance tuning—for example, comparing the impact of different batch sizes in vLLM (e.g., 16 vs. 64). It runs on Linux and macOS, is open-source (Apache 2.0), and is maintained by the same author as wrk. Note: wrk2 does not support Windows natively.

6. Apache JMeter

Apache JMeter is a full-featured, GUI-based load-testing tool written in Java. Its HTTP Request sampler can send POST requests with JSON payloads to LLM endpoints, and the View Results Tree listener shows full request/response pairs for debugging. JMeter supports correlation via regular expression extractors—you can capture a session token from the first response and reuse it in subsequent requests.

For LLM-specific testing, JMeter’s Constant Throughput Timer maintains a target RPS, and the jp@gc - Throughput Shaping Timer plugin provides more advanced load profiles (e.g., ramp-up, peak, ramp-down). JMeter is best for enterprise teams that need a GUI for test creation and integration with BlazeMeter for cloud-scale testing. It is free under the Apache 2.0 license but requires Java 8+. The GUI can be slow for tests with 10,000+ threads; use CLI mode for production runs.

7. Artillery

Artillery is a Node.js-based load-testing tool with YAML configuration for defining scenarios. It supports HTTP, WebSocket, and Socket.io protocols, making it suitable for streaming LLM responses. You can define a scenario that sends a prompt, waits for the first token, and measures time-to-first-token using custom afterResponse hooks. Artillery’s metrics include http.response_time, http.requests_per_second, and vusers.created.

Artillery is best for teams already using Node.js in their stack, as the plugin ecosystem (e.g., artillery-plugin-metrics-by-endpoint) is easy to extend. The Artillery Cloud tier starts at $0.10 per VU-hour and provides real-time dashboards. The open-source version is free under the MPL-2.0 license. For LLM testing, Artillery’s variable think time feature helps simulate human-like pauses between requests.

8. Gatling

Gatling is a high-performance load-testing tool based on Scala and Akka. It uses a DSL (Domain-Specific Language) for defining scenarios, which compiles to efficient JVM bytecode. For LLM inference, you can write a scenario that sends a POST request, extracts the response body, and checks that the token count exceeds a minimum threshold. Gatling’s HTML reports include interactive charts for response time distribution, active users over time, and errors per second.

Gatling supports gRPC via a third-party plugin and integrates with Jenkins and Grafana. It is best for Java/Scala teams that need enterprise-grade reporting and 100,000+ concurrent users on a single machine. The open-source version is free under the Apache 2.0 license; Gatling FrontLine (enterprise) starts at $1,500 per year. Gatling requires Java 11+ and has a steeper learning curve than Locust.

9. Siege

Siege is a veteran HTTP load-testing tool written in C. It supports concurrent users, delay time, and benchmark mode. You can run siege -c 50 -t 60s -f urls.txt where urls.txt contains multiple LLM endpoint URLs with different prompts. Siege outputs availability, throughput, elapsed time, and data transferred. It is best for quick sanity checks on a single server—for example, verifying that your LLM service can handle 100 concurrent requests without crashing.

Siege is not suitable for complex scenarios like streaming or custom metrics. It is open-source (GPLv2) and runs on Linux, macOS, and Windows (via Cygwin). The tool has not been actively updated since 2022, but it remains reliable for basic load testing. For teams that need a no-fuss, single-command tool, Siege is a solid choice.

10. Oha

Oha is a modern HTTP load-testing tool written in Rust, designed for low resource usage and high concurrency. It uses tokio async runtime to handle 10,000+ connections with minimal CPU overhead. For LLM inference, Oha supports custom headers and JSON body via the -b flag. Its output includes latency percentiles (p50, p90, p99), requests per second, and transfer rate in a color-coded terminal table.

Oha is best for resource-constrained environments—for example, running on a Raspberry Pi or a low-cost cloud VM. It is open-source (MIT) and compiles to a single binary. Oha does not support distributed testing or streaming responses, but it is perfect for single-node throughput tests where you need a modern, fast tool. It has 2,500+ GitHub stars and is actively maintained.

FAQ

What is the most important metric to track when load-testing an LLM service? Time-to-first-token (TTFT) and throughput in tokens per second are the two most critical metrics. TTFT measures perceived latency for the user, while tokens-per-second determines how fast the model generates responses. Most tools (Locust, k6) can capture these via custom metrics.

How many concurrent users should I simulate for a realistic test? Start with 10% of your expected peak users and ramp up by 20 users every 30 seconds until you see error rates exceed 5%. For a production LLM service expected to handle 1,000 concurrent users, begin with 100 and scale to 1,200. Use Locust’s step load pattern for this.

Can I test streaming responses with these tools? Yes, but only Locust, k6, and Artillery support streaming natively. For Locust, use the stream=True parameter in your HTTP client. For k6, use the http.post function with responseType: 'stream'. Other tools like Hey and Vegeta will wait for the full response, which inflates latency numbers.

What is the best tool for testing gRPC-based LLM servers like Triton? k6 has native gRPC support via the k6/net/grpc module. You can define proto files and call RPC methods directly. Locust requires a custom gRPC client (e.g., grpcio). Vegeta has a gRPC plugin, but it is less mature.

How do I avoid rate limiting during load tests? Use constant pacing (e.g., one request per 2 seconds per virtual user) instead of firing requests as fast as possible. In Locust, set wait_time = between(1, 3). In k6, use sleep(2) between iterations. This mimics real user behavior and prevents false positives from server-side throttling.

Can I test multiple model sizes in one test? Yes, by using parameterized prompts in your test script. For example, in Locust, you can define a task that randomly selects from a list of prompts with different max_tokens values. This helps you understand how the service behaves under mixed workloads.

What hardware should I use for the load generator? For 1,000 concurrent users, a c6i.large EC2 instance (2 vCPUs, 4 GB RAM) is sufficient. For 10,000+ users, use a c6i.4xlarge (16 vCPUs, 32 GB RAM) or distribute across multiple instances with Locust’s master/worker mode. Always run the load generator on a separate machine from the LLM server to avoid resource contention.

How do I measure token-level latency? Parse the response body to extract the token count (often in the response headers or body). Then divide total response time by token count to get time-per-token. In Locust, use a custom on_response callback. In k6, use a Trend metric.

What is the best free tool for a single developer? Hey is the best free tool for a single developer. It requires no setup, runs on any OS, and gives you latency percentiles and throughput in seconds. For more advanced needs, Locust is free and has a larger community.

How do I test WebSocket-based streaming (e.g., for GPT-4o)? Use k6 with the k6/ws module. You can open a WebSocket connection, send a prompt, and measure the time between each message. Artillery also supports WebSocket via its websocket engine. Locust does not natively support WebSocket.

Bottom Line

For most teams, Locust offers the best balance of flexibility, real-time metrics, and distributed testing for LLM inference services. If you are embedded in a Kubernetes ecosystem, k6 is a strong alternative with superior CI/CD integration. For quick, cost-free checks, Hey provides instant throughput and latency data. Always pair your load test with vLLM or TGI server logs to correlate client-side metrics with server-side performance. Start with a warm-up phase, monitor GPU utilization, and use a production-like prompt dataset for realistic results.

Related on PULSE

• [What causes high latency in LLM inference and how do you fix it?](/knowledge/ai389)
• [The 10 Best LLM Inference Servers in 2027](/knowledge/ai342)
• [How do you scale LLM inference to handle thousands of concurrent users?](/knowledge/ai347)
• [What is the difference between vLLM, TGI, and Triton for LLM inference?](/knowledge/ai345)
• [The 10 Best LLM Quantization and Inference Optimization Tools in 2027](/knowledge/ai388)
• [How do you choose an inference accelerator: GPU, TPU, or custom silicon?](/knowledge/ai415)

Sources

• Locust official documentation
• k6 official documentation
• Hey GitHub repository
• Vegeta GitHub repository
• wrk2 GitHub repository
• Apache JMeter official site
• Artillery official documentation
• Gatling official site
• Oha GitHub repository
• vLLM official documentation

*How to load-test an LLM inference service using Locust, k6, Hey, Vegeta, wrk2, JMeter, Artillery, Gatling, Siege, and Oha for latency, throughput, and token-level metrics.*