Inference engines: vLLM, TGI, SGLang

Deploying an LLM is not the same as running one locally. A production inference server must handle hundreds of concurrent requests, minimize latency, maximize GPU utilization, and support features like streaming, structured output, and model quantization. Several open-source frameworks have emerged to solve this problem.

vLLM

vLLM (Kwon et al., UC Berkeley, 2023) introduced PagedAttention and continuous batching as open-source software[1]. It is the most widely used open-source inference framework.

Architecture:

• Scheduler: manages request queues, priorities, and prefill/decode batching.
• Worker processes: one per GPU; each runs the model forward pass.
• Block manager: allocates and frees physical KV cache blocks; maintains block tables per request.
• Attention kernels: custom CUDA kernels implementing PagedAttention's block-based K/V access pattern.

Key features:

• Tensor parallelism: split attention heads across GPUs (pipeline and data-parallel too).
• Speculative decoding with configurable draft model or n-gram draft.
• Quantization: GPTQ, AWQ, FP8.
• Prefix caching: automatic detection and sharing of common prompt prefixes.
• Structured output: regex-constrained decoding and JSON schema enforcement via Outlines.

API: OpenAI-compatible REST API, making it a drop-in replacement for the OpenAI API for self-hosted models.

TGI (Text Generation Inference)

TGI (Hugging Face) is the inference server used to power the Hugging Face Inference API.

Key features:

• Flash Attention 2 integration for memory-efficient attention[2].
• Paged Attention (added after vLLM).
• Continuous batching with token streaming.
• Quantization: GPTQ, AWQ, bitsandbytes (8-bit, 4-bit).
• Custom Rust router for request management and load balancing.
• First-class support for Hugging Face Hub model loading.

Differentiator: tighter HuggingFace ecosystem integration; easier to deploy models directly from the Hub without format conversion.

SGLang

SGLang (Zheng et al., UC Berkeley, 2024) focuses on structured generation and complex LLM programs.

Key features:

• RadixAttention: extends PagedAttention's prefix caching with a radix tree structure, automatically detecting and sharing any common prefix (not just the full prompt). This is especially powerful for applications with many few-shot examples, tool definitions, or shared system prompts.
• Structured output: first-class support for constrained generation (JSON, regex, grammar) with efficient incremental scheduling.
• Multi-call programs: the SGLang DSL lets users express multi-call LLM pipelines (agents, tree-of-thought, multi-turn) as Python programs, with automatic batching and KV cache reuse across calls.
• Speculative decoding: EAGLE-style draft heads.

Use case: applications that run complex LLM programs with branching, tool calls, and structured outputs benefit most from SGLang's scheduling primitives.

TensorRT-LLM

TensorRT-LLM (NVIDIA) is a closed-source framework that compiles LLM graphs to highly optimized CUDA kernels for NVIDIA GPUs.

• Fused attention kernels optimized per GPU architecture.
• INT8/INT4/FP8 quantization with NVIDIA-optimized compute.
• In-flight batching, speculative decoding.
• Strongest throughput on NVIDIA hardware, especially H100/H200.
• Less flexibility than vLLM for experimentation.

Feature comparison

Feature	vLLM	TGI	SGLang
PagedAttention	Yes	Yes	Yes (RadixAttention)
Continuous batching	Yes	Yes	Yes
Prefix caching	Yes	Partial	Yes (radix tree)
Speculative decoding	Yes	No	Yes (EAGLE)
Structured output	Via Outlines	Limited	Native
Multi-GPU TP/PP	Yes	Yes	Yes
OpenAI-compatible API	Yes	Yes	Yes
Primary use case	General serving	HF ecosystem	Structured programs

Choosing a framework

vLLM: best default for most use cases. Well-maintained, wide model support, active community.

TGI: preferred when deeply integrated with HuggingFace Hub; better for quick deployment of new HF models.

SGLang: preferred for complex LLM programs, agent frameworks, or applications with heavy structured output requirements.

TensorRT-LLM: preferred for maximum throughput on NVIDIA hardware in production at scale, where the engineering overhead of compilation is justified.

Exercises

Exercise 1. A team serves an agent framework whose every call begins with the same large block of tool definitions and few-shot examples, followed by a short per-call query. Which prefix-caching mechanism, vLLM's full-prompt prefix caching or SGLang's RadixAttention, reuses more KV cache here, and why?

Solution

Step 1. vLLM's prefix caching detects and shares a prefix when requests share the same leading tokens. If every call's prompt starts with the identical tool-definition block, that common head is shared, but any difference earlier in the prompt (a per-call system field, a reordered example) breaks the match for everything after it.

Step 2. RadixAttention stores all active prompts in a radix tree keyed by token sequence, so it shares any common prefix branch, not just a single full-prompt prefix, and it does so automatically as prompts fork. With many calls that share tool definitions and few-shot examples but diverge at the query, the radix tree keeps the shared branch resident once and grows only the divergent leaves.

Step 3. RadixAttention reuses more cache here. The workload (heavy shared context, short divergent tails, many concurrent calls) is exactly the structured-program case SGLang targets, which is why it is the recommended engine for agent and tree-of-thought pipelines.

Exercise 2. From the feature comparison, a deployment needs speculative decoding, native JSON-schema-constrained output, and prefix sharing across many few-shot calls. Walk through which of vLLM, TGI, and SGLang satisfy all three, and name the single disqualifying gap for each that fails.

Solution

Step 1. Requirement-check TGI: it has continuous batching and paged attention, but the comparison lists speculative decoding as "No" and structured output as "Limited". TGI fails on speculative decoding (and structured output).

Step 2. Check vLLM: speculative decoding "Yes", prefix caching "Yes", but structured output is "Via Outlines", an add-on rather than native scheduling. It can do all three functionally, with constrained decoding bolted on rather than integrated into the scheduler.

Step 3. Check SGLang: speculative decoding "Yes (EAGLE)", structured output "Native", prefix caching "Yes (radix tree)". It satisfies all three first-class. SGLang is the fit; vLLM is workable with the Outlines caveat; TGI is disqualified by the missing speculative decoding.

Exercise 3. TensorRT-LLM compiles the model graph to GPU-specific kernels ahead of time, while vLLM runs a flexible Python-level engine. Describe one workload where the compilation overhead pays off and one where vLLM's flexibility is worth more, and state the underlying tradeoff.

Solution

Step 1. Compilation pays off for a stable, high-volume production endpoint: a fixed model on H100/H200 serving millions of requests, where the one-time cost of compiling fused, architecture-tuned kernels is amortized over enormous throughput, and TensorRT-LLM's per-GPU kernel optimization extracts the most tokens per second.

Step 2. vLLM's flexibility wins during experimentation and frequent model churn: swapping in new Hugging Face checkpoints, trying quantization schemes, or changing decoding strategies, where recompiling for every change is prohibitive and the Python-level engine adapts immediately.

Step 3. The tradeoff is peak throughput versus iteration speed. Ahead-of-time compilation buys maximal hardware utilization at the cost of rigidity; a flexible runtime buys fast change at the cost of leaving some throughput on the table. The right choice tracks how stable the deployment is.

The inference engine is invisible to end users but determines the cost and latency of every API call. The throughput these engines extract rests on the partitioning and memory-traffic analysis of large-model inference[3].

---

References

1. [1]Kwon, W., et al. (2023). "Efficient memory management for large language model serving with PagedAttention". SOSP. arXiv:2309.06180. The vLLM system paper.↩
2. [2]Dao, T., Fu, D., Ermon, S., Rudra, A., & Ré, C. (2022). "FlashAttention: fast and memory-efficient exact attention with IO-awareness". NeurIPS. arXiv:2205.14135. The IO-aware fused attention kernel these servers build on.↩
3. [3]Pope, R., et al. (2023). "Efficiently scaling transformer inference". MLSys. arXiv:2211.05102. Partitioning strategies and the latency/throughput tradeoffs of serving large transformers.↩