If you’ve been using ray-project/llmperf, you may have noticed it’s now in archive mode. No new updates, no fixes, no responses to issues. If you’re evaluating it for the first time, that’s worth knowing before you build anything on top of it.

This page is part of my LLM benchmarking guide, which covers the metrics themselves (TTFT, ITL, throughput). Here I want to focus on the tools — what’s out there now that llmperf is effectively done, and what I ended up building.

What llmperf was good at #

Credit where it’s due — llmperf was the go-to open-source option for benchmarking OpenAI-compatible endpoints. It measured the metrics that matter (TTFT, ITL, throughput), handled concurrency, and came out of the Anyscale/Ray team, so it had credibility. For a lot of teams it did the job well.

Why I started looking around #

Two things pushed me to look at alternatives, and both are about fit rather than flaws:

1. ITL aggregation. llmperf computes Inter-Token Latency by averaging within each request first, then aggregating those per-request averages. That’s a reasonable choice and works well for many use cases. But I was specifically trying to catch latency spikes during the decode phase, and per-request averaging smooths exactly those out. I needed the raw distribution. Why this matters is covered in the main guide.

2. Startup overhead. Runs spawn Ray workers, so there’s a meaningful spin-up cost before the first request fires. When I just want to poke at an endpoint quickly, that’s more ceremony than I want — I was after something closer to curl than a cluster setup.

Neither is a defect. They’re design decisions that matched llmperf’s goals and didn’t match mine. So I looked at what else exists.

The alternatives #

A note on genai-perf: NVIDIA sunsetted it and moved development to aiperf. I’d tried genai-perf earlier and couldn’t get it running on Ubuntu 22.04 without a Docker container, did not try the source install.

When to use which #

Pick based on what you’re measuring:

• You need GPU-level metrics (prefix caching, kernel-level breakdown) → aiperf or trtllm-bench. This is where llmperf-rs won’t help you.
• You’re testing vLLM-specific behavior → vllm-bench.
• You want dashboards and visual reporting → GuideLLM.
• You need distributed load generation → aiperf.
• You just want to hit an OpenAI-compatible endpoint quickly, with minimal setup, and see TTFT/ITL/throughput → llmperf-rs.

That last one is the gap I was trying to fill — something I could drop onto a box and run in seconds, that preserved raw ITL values so spikes weren’t hidden.

What I built: llmperf-rs #

llmperf-rs is a single Rust binary that benchmarks any OpenAI-compatible endpoint (vLLM, Ollama, local APIs). It:

• Keeps raw ITL values across all responses before computing percentiles, so p99 and max actually reflect spikes.
• Uses API-reported token counts from the usage field when available, falling back to a tokenizer you specify. The original llmperf used one tokenizer for everything, which gets inaccurate across model families with very different vocab sizes.
• Outputs console summaries plus JSON for digging in with pandas.

It’s not trying to compete with the GPU-deep tools. I think of it as one level above curl — fast to start, low dependency, good enough for most “how’s this endpoint doing” questions. The trade-off is no GPU metrics, no distributed testing.

If that trade-off sounds right for you: grab it from the releases page, or cargo install --git https://github.com/wheynelau/llmperf-rs.

The takeaway #

llmperf being archived doesn’t mean the tooling disappeared — aiperf, vllm-bench, GuideLLM, and others are all actively maintained. The choice mostly comes down to how deep you need to go (GPU metrics vs endpoint metrics) and how much setup you’re willing to tolerate.

For the metrics themselves — what TTFT, ITL, and throughput actually tell you, and where they mislead — see the main benchmarking guide.