Why the engine choice matters

The inference engine decides how efficiently your GPU serves tokens. The same model on the same hardware can deliver very different throughput, latency, and cost depending on the engine and how its batching and caching are tuned.

There is no universal winner. vLLM, SGLang, and TensorRT-LLM are all actively developed and converge on many techniques, including continuous batching, paged KV cache, prefix caching, and quantization. The right pick depends on your model, GPU, sequence lengths, batch size, and concurrency.

vLLM: the high-throughput default

vLLM is built around PagedAttention, which manages the attention key value cache in non contiguous pages the way an operating system manages virtual memory. That reduces memory fragmentation and lets the server pack more concurrent sequences onto a single GPU.

It is a strong, general purpose choice for high throughput serving with continuous batching, broad model coverage, and an easy OpenAI compatible server. Pick it when you want wide model support and solid throughput without a compilation step. For a deeper look, see our vLLM explainer.

SGLang: built for prefix reuse and structured outputs

SGLang is built around RadixAttention, which reuses the KV cache across requests that share a prefix using a radix tree. That benefits workloads with heavy prompt sharing, such as few shot prompts, long shared system prompts, agentic multi turn conversations, and tree of thought.

It also ships fast, reliable structured outputs, including JSON, regex, and grammar constrained decoding, which helps complex prompting and tool calling pipelines. Reach for SGLang when prefix reuse and constrained generation dominate your traffic.

Not sure which engine fits your workload?

We will benchmark vLLM, SGLang, and TensorRT-LLM on your model and GPU, then put the winner behind one gateway.

Book a demo

TensorRT-LLM: maximum optimization on NVIDIA GPUs

TensorRT-LLM is NVIDIA's library that compiles models into optimized engines for a specific GPU target, using fused kernels, optimized attention, quantization on supported hardware, and in flight batching through Triton.

‍

It typically aims for the lowest latency and best per GPU efficiency on NVIDIA GPUs, at the cost of an explicit build step and engines that are tied to the exact GPU type and count they were built for. As the docs note, since TRT-LLM optimizes the model for the target GPU type and counts, it is important that the GPU type and count matches while deploying.

‍

At a glance

‍

| Engine | Core idea | Best for | Tradeoff |

| --- | --- | --- | --- |

| vLLM | PagedAttention KV cache | General high throughput, broad model support | Great default, not always lowest latency |

| SGLang | RadixAttention prefix reuse | Shared prompts, agentic, structured outputs | Biggest wins need prefix reuse |

| TensorRT-LLM | Compiled NVIDIA engines | Lowest latency on NVIDIA GPUs | Build step, engine tied to GPU |

‍

Any performance numbers are workload specific. Benchmark on your own traffic before committing. TrueFoundry supports all three so the engine can be matched to the workload.

‍

Deploying vLLM or SGLang on TrueFoundry

‍

For vLLM and SGLang, the path is the model catalogue. As the docs say, TrueFoundry simplifies the process of deploying an LLM by automatically figuring out the most optimal way of deploying it and configuring the correct set of GPUs.

‍

You can either choose one of the models in the LLM catalogue or paste a Hugging Face model URL. TrueFoundry analyzes the model tags and files to understand the deployment framework and generates the deployment manifests. You keep complete flexibility in choosing the most optimal model server, which guarantees faster inference. The platform also uses image streaming and caching, which gives 3X faster download times for vLLM and SGLang images.

‍

For gated models, attach your Hugging Face token as a stored secret in the secret picker.

‍

Deploying TensorRT-LLM on TrueFoundry

‍

TensorRT-LLM is not a single click toggle, because engines must be compiled for the target GPU. The dedicated guide uses a build then serve flow.

‍

• Add your Hugging Face token as a Secret for gated models, then create an ML Repo and a workspace with access to it. The ML Repo stores and versions the compiled engines.
• Deploy the engine builder job using the TrueFoundry engine builder image, setting parameters such as model id, dtype, max batch size, and max input length. When it finishes, the engine appears under Models and the tokenizer under Artifacts.
• Deploy with NVIDIA Triton using the Triton server image, pointing at the engine and tokenizer FQNs. The GPU type and count must match the builder job.
• Run inferences against an OpenAI compatible endpoint. Since the endpoint is OpenAI compatible, you can add it to the AI Gateway or use it directly with the OpenAI SDK.

‍

Running inference against a deployed TensorRT-LLM engine through the OpenAI compatible endpoint.

‍