vLLM Benchmark: Qwen3-8B vs Llama 3.1 8B vs Ministral 8B on a single A10

We ran GuideLLM concurrent sweeps against three 8B-parameter models on a single NVIDIA A10 (24 GB) GPU: Qwen3-8B, Llama 3.1 8B Instruct, and Ministral 8B. This vLLM benchmark swept concurrency from 4 to 256 simultaneous users across two workloads: short conversational chat and intent classification. One model held up across the entire range. The other two didn't.

The setup

All three models ran FP16 on vLLM, served through the standard /v1/chat/completions streaming endpoint. GuideLLM held exactly N requests in-flight for 90 seconds per concurrency level (4, 8, 16, 32, 64, 128, 256), with 30 seconds of cooldown between levels.

Two workloads:

• Short chat: approximately 180 input tokens, 130-150 output tokens. Reflects a copilot or conversational assistant use case.
• Classifier: approximately 60 input tokens, 4 output tokens. Reflects an intent router or ticket classifier.

Same vLLM version, same gateway config, same GPU for all three. The results below reflect those conditions.

Results at a glance

The summary table uses 64 concurrent users for the short chat comparison. That is the last concurrency level where Llama ran clean (0% errors), so it is the fairest common baseline across all three models.

Qwen3 has the lowest typical wait (524 ms p50) and the tightest tail (2,840 ms p95). Llama pushes 11% more tokens per second, but the median user waits 2,569 ms for the first token compared to Qwen3's 524 ms. One thing stood out in the data: Llama's E2E p95 (10.4 s) is lower than Qwen3's (11.4 s) even though its TTFT is much worse. Llama starts slow but decodes fast. Once tokens start flowing, they flow faster than on Qwen3. For streaming UIs, though, that first-token delay is what users actually feel.

Full percentile breakdowns at every concurrency level, Poisson burst-traffic results, and downloadable CSVs are in the interactive benchmark report.

Short chat: who breaks first under concurrency

At low concurrency, all three models started at roughly the same TTFT p50, around 350 ms. Qwen3 stayed there. At 16 concurrent users, Qwen3's p50 was 388 ms. Llama's had already climbed to 794 ms.

Qwen3-8B: zero errors at every level

Qwen3 completed every request at every concurrency level through 256. No dropped requests. TTFT p50 at 64 concurrent was 524 ms with 851 tok/s throughput.

Our working hypothesis for why it holds up under load: Qwen3 uses QK-LayerNorm, which stabilizes attention scores under long KV caches, and its slightly smaller per-layer KV footprint gives vLLM more scheduling headroom before requests start queuing at the gateway. Llama's larger FFN (14,336 vs Qwen3's 12,288) drives higher per-token throughput but fills the decode pipeline faster, leaving less room for new prefills as concurrency climbs. We're still investigating, but the architecture difference appears to matter here.

Llama 3.1 8B: highest throughput, collapses at 128

Llama posted the highest raw throughput at peak: 1,181 tok/s at concurrency 256. At concurrency 128, however, 86.9% of requests got 408 timeouts. The 1,181 tok/s figure comes from the few requests that made it through after the server had already shed most of its queue.

At 64 concurrent (the last clean point), Llama delivered 947 tok/s with TTFT p95 of 4,062 ms and 0% errors. That's 11% more throughput than Qwen3 at the same load. For batch inference or async pipelines where retries are acceptable, that throughput advantage is real. For interactive use, cap it at 32-64 concurrent.

Ministral 8B: persistent errors across the whole range

We have to flag this honestly. Ministral showed 3.8% real HTTP errors at concurrency 4, the lightest load we tested. At 64 concurrent it was 5.3%. At 128 it reached 8.9%. These were genuine 408/503 failures, not benchmark artefacts. This probably reflects deployment instability during our test window rather than a fundamental model limitation, but the data is the data.

Where requests succeeded, Ministral's latency was competitive: TTFT p50 of 350 ms at baseline and 512 ms at concurrency 64, close to Qwen3's 524 ms. Latency numbers don't mean much, though, if roughly 1 in 20 requests never gets a response.

Classification: all three reached 256 concurrent, but latency curves split

For the classifier workload, all three models ran clean through 256 concurrent users with zero errors. Llama led on raw requests per second (173 RPS), followed by Qwen3 (157 RPS) and Ministral (143 RPS).

The more telling number is how much latency moved under load. Qwen3's TTFT p95 at 128 concurrent was 541 ms, only 32% above its baseline of 409 ms. The p50 moved from 358 ms to 467 ms. Llama's p95 nearly doubled (351 ms to 689 ms). Ministral's more than doubled (390 ms to 823 ms).

If you need an intent router or classifier with predictable latency under bursty traffic, Qwen3's flat curve is worth paying attention to. The GPU absorbs short classifier requests into a single batch without queuing, and that shows in the numbers.

Throughput scaling across the sweep

All three start in the same place and scale linearly through 16 concurrent users. The gap opens past that.

Past 64, Llama's throughput keeps climbing on paper (1,181 tok/s at 256) but most requests are being rejected. Qwen3 plateaus around 830-850 tok/s and serves every request. Ministral falls about 15% short of Qwen3 at 64 concurrent even among the requests that succeeded.

The raw peak number without an error rate next to it is close to meaningless for capacity planning. All three of these vLLM benchmarks illustrate that.

Which model to deploy

• Chat and copilots: Deploy Qwen3-8B. It doesn't drop requests, scales gracefully, and TTFT is competitive. Cap concurrency at 64 for interactive use (TTFT p95 stays under 3 seconds) or push higher for async workloads.
• Batch inference: Llama 3.1 8B, capped at 64 concurrent. Past that, the gateway sheds requests aggressively. Best choice when you can retry and want maximum tokens per GPU-second.
• Classification: Any of the three work at scale. All ran clean to 256 concurrent. Llama is roughly 10% faster on raw RPS. Qwen3 has the flattest latency under load.
• Ministral 8B: Re-test on a clean deployment before drawing conclusions. The persistent low-concurrency errors in chat make this run's data unreliable for streaming workloads.
  

How to serve these models with TrueFoundry

We ran this vLLM benchmark on a single A10 spot instance deployed via TrueFoundry. The full vLLM config and GuideLLM commands will be published with the deep-dive post.

TrueFoundry's model serving layer deploys vLLM backends on any GPU tier (A10, A100, H100) directly from the control plane. You pick the model, set max_model_len and tensor_parallel_size, and TrueFoundry handles the container, the GPU allocation, and the health checks. The model comes up on a standard OpenAI-compatible endpoint.

In front of vLLM, TrueFoundry's AI Gateway adds roughly 3-4 ms of overhead while handling 350+ RPS on a single vCPU. It gives you per-model rate limits, cost attribution, fallback routing across providers, and a full request trace. If you're comparing multiple models (as in this benchmark), you can route all three through the same gateway endpoint and switch between them by changing the model field in the request. The URL and credentials stay the same.

TrueFoundry supports 1,000+ LLMs through a single OpenAI-compatible API, so moving from a benchmarked open-weight model to a commercial provider doesn't require any client-side changes.

For teams self-hosting open-weight models behind a gateway, this architecture keeps the serving layer separate from routing and governance. See 10 Best LLM Observability Tools in 2026 for monitoring your vLLM deployment once it's running.

Related reading

• Top 5 LiteLLM Alternatives in 2026 — if you're evaluating gateway options alongside model serving
• 10 Best LLM Observability Tools in 2026 — monitoring your vLLM deployment in production
  

Conclusion

This vLLM benchmark makes one thing clear: throughput numbers need error rates next to them to mean anything. Llama 3.1 8B's 1,181 tok/s peak looks impressive until you see that 87% of requests at that load timed out. Qwen3-8B's 851 tok/s at 64 concurrent, with zero failures through 256, is the number that matters for production planning.

For 8B model selection on a single A10, the data points toward Qwen3 for interactive workloads and Llama for batch jobs where you control concurrency and can handle retries. Ministral needs a clean retest before any production recommendation.