Inference Infrastructure — Swarnim Tiwari

01

How is each framework architected?

An LLM inference framework is not a model runner. It is a system that manages GPU memory, schedules thousands of concurrent requests, handles two fundamentally different computational phases, and tries to keep expensive hardware near 100% utilization at all times. The architectural choices made to solve that problem are what separate these four frameworks.

View

Python · Apache 2.0 · v0.23.0 (June 2026)

vLLM

The Production Workhorse

Architecture

PagedAttention (KV paging)

v1 engine (modular, 2026)

NVIDIA + AMD + TPU + Intel

OpenAI compatible API

Python first, extensible

01UC Berkeley origin, Apache 2.0. The PagedAttention paper published in September 2023 reframed how the field thinks about KV cache: instead of reserving contiguous memory blocks per sequence, vLLM allocates cache in fixed-size pages that can occupy non-contiguous physical memory. This single change transformed GPU memory utilization from 20 to 40% efficiency to over 96%, enabling batch sizes that were previously impossible on the same hardware.
02The v1 engine introduced in 2025 and now the default as of 2026 separates the scheduler, model executor, and detokenizer into distinct, non-blocking layers. The scheduler decides which requests to process next. The executor runs the GPU kernel. The detokenizer converts token IDs to text. These three things used to run sequentially in a single loop. Separating them means each layer can work on different requests simultaneously without waiting on the others.
03Broadest hardware support in the comparison by a significant margin. NVIDIA (all A-series, H-series, and Blackwell), AMD (MI300X, MI325X, MI355X) via ROCm, Google TPU via XLA, Intel Gaudi via HPU, and CPU fallback. The AMD path specifically matured through 2025 and is production-ready as of June 2026 with ROCm 6.3 and newer. This is why teams building for multi-vendor GPU fleets or using AMD hardware choose vLLM before any benchmark is run.
04Community scale as of June 2026: vLLM 0.23.0 is stable, the project has thousands of contributors, and new model architectures are typically supported within days of public release. TensorRT-LLM takes weeks to validate new models through its compilation toolchain. SGLang is faster than TRT but slower than vLLM. For teams deploying the latest released model the day it ships, vLLM is the only reliable path.
05Prefix caching in vLLM uses hash-based exact match: if the prefix of a new request exactly matches a previously cached sequence, those KV pages are reused. This works well for identical repeated system prompts and fixed instruction blocks. It does not handle the case where two requests share a partial but non-identical prefix. SGLang's RadixAttention handles this case. For workloads with strict identical prefixes, the two approaches perform equivalently. For workloads with overlapping but varied prefixes, SGLang outperforms vLLM on cache efficiency.

vLLM is the right answer when you do not yet know which answer is right. Broadest hardware support, fastest new model adoption, largest community, most mature documentation. The performance gap versus SGLang on cache-heavy workloads and versus TRT-LLM on raw NVIDIA throughput is real but not large enough to justify choosing a less battle-tested framework unless your specific workload profile clearly favors one of them.

Python · Apache 2.0 · v0.5.13 (June 2026)

SGLang

Structured Generation and Shared Prefix

Architecture

RadixAttention (trie KV tree)

Zero overhead CPU scheduler

xgrammar (structured output)

Disaggregated prefill-decode

PyTorch native (March 2025)

01LMSYS project from the team behind Chatbot Arena. Apache 2.0. v0.5.13 as of June 2026. Integrated into the official PyTorch ecosystem in March 2025, which means enterprises standardized on PyTorch infrastructure can adopt SGLang without treating it as an external dependency. This PyTorch integration was a deliberate signal that SGLang is not an academic experiment — it is production infrastructure.
02RadixAttention is the architectural differentiator. Where vLLM's prefix cache stores flat hash maps of exact prefix matches, SGLang builds a radix tree (also called a trie) over all cached KV pages across all currently serving requests. When a new request arrives, the engine traverses the tree to find the longest matching path across every previously cached sequence. Partial prefix sharing is handled automatically. Two requests sharing only their first 500 tokens of context both benefit from the cached portion without any developer configuration.
03Zero overhead CPU scheduler: the scheduling thread assembles the next batch on CPU while the GPU executes the current batch. In earlier vLLM versions and in TGI, the scheduler occasionally blocked GPU execution while deciding which requests to process next. SGLang designed the scheduling path to always have the next batch ready before the GPU completes the current step. Under high concurrency, this eliminates a source of GPU idle time that compounds across millions of requests.
04Structured generation is native via xgrammar, not an external integration. JSON schema enforcement, regular expression constraints, and tool call format validation happen at the token level during the decode loop, running on GPU inside the same kernel as decoding. Frameworks that integrate Outlines or similar external libraries for structured output run constraint validation on CPU between GPU decode steps. The overhead difference is measurable at scale: SGLang benchmarks show under 5% latency overhead for JSON schema enforcement versus unconstrained generation.
05In a February 2026 collaboration between LMSYS and NVIDIA, SGLang running DeepSeek R1 on a GB300 NVL72 rack-scale system achieved 25x higher performance compared to running the same model on H200. This number reflects both the Blackwell hardware generation improvement and SGLang's optimizations for the new memory and interconnect architecture. For MoE models (Mixture of Experts) like DeepSeek and Mixtral, SGLang's expert parallelism and routing-aware scheduling produces higher throughput than vLLM's general-purpose scheduler on equivalent hardware.

SGLang wins when your workload has prefix sharing and when you need structured output at production scale. RAG systems, multi-turn conversational applications, and agentic workflows that share context across calls are all SGLang-native workloads. The RadixAttention advantage on those workloads is not a marginal benchmark improvement. At 70% cache hit rate, you are serving roughly three times more requests per GPU compared to no caching. That is a real infrastructure cost difference.

C++ / Python · Apache 2.0 · v1.2.1 (June 2026) · NVIDIA Only

TensorRT-LLM

Compiled Engine, NVIDIA Peak

Architecture

Compiled TensorRT engine

Kernel fusion and CUDA graphs

NVFP4 on Blackwell

NIM container deployment

Dynamo integration

01NVIDIA's own serving framework. Apache 2.0 but NVIDIA hardware only — this is a hard architectural boundary, not a configuration limitation. The core concept is compilation: before serving a model you build an optimized TensorRT engine that fuses operations, eliminates Python runtime overhead from the critical path, and captures CUDA execution graphs for every batch configuration. This compilation produces faster execution at the cost of build time and configuration flexibility.
02Kernel fusion is the primary source of TRT-LLM's raw performance advantage. In a standard Python inference framework, attention computation, KV cache update, and residual connection are three separate GPU kernel launches. TRT-LLM fuses them into a single CUDA kernel, eliminating the CPU-to-GPU synchronization overhead between launches. At high decode step rates on fast hardware, kernel launch overhead represents a meaningful fraction of total step time. CUDA graph capture eliminates the remaining launch overhead entirely.
03TRT-LLM v1.2.1 introduces EAGLE3 speculative decoding support and deepens Blackwell integration. The Blackwell GB200 and GB300 GPU architecture provides new tensor core designs specifically optimized for NVFP4 (4-bit float with microscaling), and TRT-LLM is the first serving framework to fully utilize those cores. On Blackwell with NVFP4, TRT-LLM reports more than double the throughput of H100 with FP8 on equivalent models, per NVIDIA internal benchmarks published in early 2026.
04The compilation constraint is the real operational cost. A configuration change that vLLM or SGLang applies at startup — different maximum sequence length, different quantization level, different parallelism degree — requires a full engine rebuild in TRT-LLM. At 70 billion parameters, a rebuild takes 30 to 60 minutes on modern hardware. For teams responding to a production performance incident at 2am, that rebuild delay is significant. For teams with stable, well-understood production traffic on fixed model configurations, the rebuild cost is a one-time expense per deployment.
05NIM (NVIDIA Inference Microservices) are prebuilt, preoptimized TRT-LLM containers for specific model and hardware combinations that deploy with a single Docker command without the build step. For teams running supported models (Llama, Mistral, Nemotron, DeepSeek, and others) on NVIDIA hardware, NIM is the fastest path to production-quality TRT-LLM serving without infrastructure engineering investment. NIM containers include serving APIs, health endpoints, and Prometheus metrics out of the box.

TRT-LLM makes sense when two conditions are simultaneously true: you are committed to NVIDIA hardware for the foreseeable future, and your model configuration is stable enough that rebuilding the engine is not a frequent operational task. Both conditions need to hold. Teams that meet both get the highest throughput and lowest latency available in open serving infrastructure on NVIDIA GPUs. Teams that fail either condition are paying the compilation complexity cost without getting the full performance benefit.

Rust · Apache 2.0 · Maintenance Mode (Late 2025)

Text Generation Inference

Maintenance Mode. Migrate for New Projects.

Architecture

Rust serving core

Flash Attention (standard)

HF Hub native integration

Continuous batching

Security patches only

01Built by Hugging Face. Rust core with a Python frontend. In maintenance mode as of late 2025: Hugging Face officially recommends vLLM or SGLang for new deployments. TGI receives security patches and critical bug fixes. No new features are being developed. This is the third volume in this series where the incumbent tool in a category entered maintenance mode during the research window. The pattern is consistent: simpler tools built for an earlier problem definition get displaced when the infrastructure complexity grows past what they were designed to handle.
02The Rust core was a genuine architectural advantage when TGI launched. Memory safety without garbage collection, predictable latency without GC pauses, and near-zero overhead for the HTTP serving layer were meaningful advantages over Python-based alternatives in 2022 and 2023. By 2025, vLLM and SGLang had closed the performance gap through PagedAttention and RadixAttention improvements that are independent of language choice. The Rust core is no longer the deciding factor in any performance comparison.
03Hugging Face ecosystem integration remains TGI's strongest feature for teams inside the HF stack. Automatic model downloading from the Hub, native support for HF model formats, direct integration with HF Datasets, and Inference Endpoints (managed TGI on AWS, Azure, or GCP through the HF Hub) all work without additional tooling. For teams whose entire workflow runs through HF infrastructure, TGI requires zero new services to evaluate against a model checkpoint.
04The production install base is substantial and rational. Teams currently running stable TGI deployments against known model configurations with acceptable throughput have made a correct calculation: migration costs engineer time and production testing risk. Staying on TGI costs nothing as long as the workload is not throughput-constrained and no new model architecture is needed that TGI does not support. Many production deployments will run TGI for years before any pressure to migrate emerges.
05For new projects starting in 2026, TGI is not the right default. The HF recommendation itself is the clearest signal: when the organization that built a tool recommends competitors for new projects, the tool's trajectory is clear. The API compatibility between TGI and vLLM means the migration path, when it becomes necessary, is a URL change on the client side and a Docker image swap on the server side.

TGI's maintenance mode status is not a critique of the tool's quality. TGI served its purpose well and processes production inference traffic across thousands of deployments as of mid-2026. Maintenance mode is a product decision, not a failure verdict. For teams running it today with stable workloads, no action is required. For new projects, the choice of TGI requires explaining why you are choosing a frozen technology when two actively developed alternatives with better performance are available and compatible with the same API surface.

02

How does each framework manage KV cache?

KV cache is the dominant memory consumer in production LLM serving. Every token the model has seen in the current context occupies memory as a key and value tensor for each attention layer. At scale, how you allocate, reuse, and evict that cache determines whether you can serve 100 concurrent users or 1,000 on the same hardware.

View

Python · Apache 2.0

vLLM

PagedAttention

KV Stack

Non-contiguous page allocation

Hash-based prefix cache

LRU eviction

CPU swap for preempted sequences

Lock-free allocator (v1)

01PagedAttention allocates KV cache in fixed-size pages that can occupy non-contiguous physical GPU memory — the same concept as virtual memory paging in an operating system. When a new sequence arrives, vLLM assigns empty pages from a shared pool. When a sequence finishes, its pages return to the pool for reuse. There is no requirement that a sequence's pages be adjacent in memory, which eliminates the external fragmentation that forced earlier systems to pre-reserve maximum context memory per slot.
02The memory efficiency improvement is documented in the original PagedAttention paper: less than 4% memory waste versus 60 to 80% waste in systems using contiguous block allocation. In practical serving terms, this means vLLM supports 2 to 4 times more concurrent sequences on the same GPU compared to naive implementations. The actual multiplier depends on sequence length distribution — workloads with highly variable lengths benefit most.
03Prefix caching stores computed KV pages from completed requests and matches them to new requests via hash comparison. A system prompt shared across 10,000 user requests computes once and reuses across all subsequent requests. Cache hits eliminate both the GPU computation and the memory allocation for the matched portion. For deployments where the system prompt is a large fraction of total token count, this produces throughput gains proportional to the prefix ratio.
04When the KV pool fills under high load, vLLM preempts lower-priority sequences via two strategies: swap (move KV pages to CPU RAM and resume the sequence when GPU space frees) or recompute (discard the pages and restart the sequence from scratch on next scheduling). Swap preserves sequence state at the cost of PCIe bandwidth for the transfer. Recompute avoids the transfer cost at the cost of wasted prior computation. The policy is configurable per deployment.
05The v1 engine's KV allocator is lockfree for the common allocation path in multi-GPU tensor parallel configurations. In the original architecture, the allocator held a Python-level lock during page assignment, which created contention at 8 GPU tensor parallel setups serving thousands of concurrent requests. The lockfree redesign removed this as a measurable bottleneck for large-scale deployments.

PagedAttention solved the memory fragmentation problem that was the binding constraint on LLM serving concurrency before 2023. The hash-based prefix cache built on top of it solves the repeated computation problem for workloads with shared system prompts. The gap versus SGLang's RadixAttention is that hash-based caching handles the simpler case. For production deployments where the same exact system prompt appears on every request, vLLM's prefix cache is fully adequate.

Python · Apache 2.0

SGLang

RadixAttention

KV Stack

Radix tree over KV pages

Partial prefix sharing

Reference-count eviction

60-95% hit rate in RAG/chat

Disaggregated prefix nodes

01RadixAttention builds a radix tree (trie) over all KV cache pages across all currently active requests. Each path from the root represents a cached token sequence. When a new request arrives, the engine traverses the tree to find the longest matching path — not just an exact match of the entire prefix but the longest common subsequence across every previously cached sequence. Partial prefix sharing is handled automatically by the tree structure without developer configuration or workload-specific tuning.
02The practical difference from vLLM's hash-based prefix cache becomes visible in RAG workloads. Consider a system where different users query against the same set of retrieved documents. The retrieved document content is identical across those queries but each user's conversation history is different. In vLLM, the document block cannot be cached independently unless it appears as the literal beginning of every sequence. In RadixAttention, the document block is a shared internal node in the tree and all queries sharing it reuse the cached computation regardless of what precedes or follows it in each individual sequence.
03Eviction uses reference counting combined with LRU. Internal tree nodes shared by multiple active sequences cannot be evicted until all sequences using them complete. Leaf nodes can be evicted immediately when no active sequence references them and the pool is under pressure. This policy preserves the highest-value shared prefixes — the ones referenced by the most concurrent sequences — for the longest time under load.
04Cache hit rates reported by SGLang in multi-turn conversational workloads: 60 to 95% of KV computation served from cache. In RAG systems where users share a consistent retrieved document corpus, similar numbers apply. The throughput gain is proportional: at 70% cache hit rate, the GPU spends 70% of its would-be compute cycles serving additional requests instead of recomputing already-known context. The number of requests served per hour increases accordingly.
05SGLang 0.5.x added disaggregated prefill support where the RadixAttention tree lives on a dedicated pool of prefill-specialized nodes. When a decode node needs to process a new request, the relevant KV pages transfer from the prefill pool over high-bandwidth interconnect. This extends the caching benefit across node boundaries in large cluster deployments and is the architecture NVIDIA Dynamo uses when orchestrating SGLang for disaggregated serving at scale.

RadixAttention represents a fundamentally different approach to the caching problem compared to hash-based exact prefix matching. It is not an incremental improvement — it handles a broader class of sharing patterns. The reason more teams do not run SGLang by default is that vLLM's simpler approach is adequate for workloads with fully identical prefixes, which is the most common simple RAG pattern. For workloads with overlapping but varied context, RadixAttention is the only cache implementation in this comparison that handles the case without manual workarounds.

C++ / Python · Apache 2.0 · NVIDIA Only

TensorRT-LLM

Compiled KV with FP8 and NVFP4

KV Stack

Paged KV in compiled kernels

FP8 KV (H100 default)

NVFP4 KV (Blackwell)

Fused attention plus KV update

Prefix caching (exact match)

01KV cache in TRT-LLM is implemented in compiled CUDA kernels rather than Python-managed allocation. The paged approach is structurally similar to vLLM's PagedAttention but the page management logic runs entirely on GPU in compiled code. This eliminates the CPU round-trips that Python-managed cache allocation requires, reducing per-step overhead at very high decode rates where every microsecond of CPU work delays the next GPU kernel launch.
02The most significant TRT-LLM KV optimization: attention computation and KV cache update fuse into a single GPU kernel during the decode phase. Standard implementations execute these as two separate kernel launches — compute attention scores using the existing cache, then write the new token's key and value vectors to the cache. Fusing them into one kernel halves the number of GPU memory round-trips on the decode path, where memory bandwidth is the dominant constraint.
03FP8 KV cache quantization is the recommended default for H100 deployments in TRT-LLM documentation as of mid-2026. Storing key and value tensors in 8-bit float rather than 16-bit halves the memory consumed by the KV cache. For a deployment serving 32,000 token context windows at high concurrency, this halved KV memory footprint is the difference between the workload fitting in GPU memory or requiring double the hardware.
04NVFP4 KV cache on Blackwell GPUs reduces memory to one quarter of FP16. Combined with FP4 weight quantization, a 70 billion parameter model in FP16 that would require approximately 140GB of weight memory fits in under 20GB. The remaining GPU memory budget can be allocated entirely to KV cache, enabling very long contexts or very large batches on a single GPU that would have required multiple GPUs at full precision.
05Prefix caching in TRT-LLM handles exact prefix matches and is functional but not optimized for partial prefix sharing. For deployments with identical repeated system prompts, performance is competitive with vLLM. For RAG workloads where document blocks are shared partially across queries with different surrounding context, TRT-LLM does not benefit from the shared portion the way SGLang's RadixAttention does. This is the primary KV cache limitation compared to SGLang for mixed-context production workloads.

TRT-LLM's KV cache implementation wins on raw memory efficiency and decode-step latency for stable NVIDIA deployments. The fused attention-plus-update kernel is a genuine optimization that other frameworks cannot match without similar compilation infrastructure. The limitation is that the optimization is baked into the compiled engine and therefore inflexible. When NVFP4 KV cache becomes standard on Blackwell hardware, TRT-LLM's compiled precision support means it arrives fully optimized for the new format faster than Python-managed alternatives.

Rust · Apache 2.0 · Maintenance Mode

Text Generation Inference

Paged KV, No Future Improvements

KV Stack

Flash Attention with paged KV

Exact prefix cache

Stable, not evolving

No RadixAttention planned

No NVFP4 KV support

01TGI uses Flash Attention with a paged KV cache implementation that was production-mature before either vLLM's PagedAttention or SGLang's RadixAttention existed. The implementation is correct and reliable. It is not the performance frontier and will not become it, because maintenance mode means no new algorithmic KV improvements are coming. Teams on TGI are running with the KV management approach from 2022 to 2023 serving LLM traffic in 2026.
02Prefix caching in TGI supports simple exact prefix reuse. Identical repeated system prompts benefit from the cache. Overlapping but non-identical prefixes do not. The implementation is correct for its design intent. The gap versus SGLang's RadixAttention is not a bug in TGI — it is a design scope difference between a tool built before RadixAttention existed and one built after.
03Flash Attention integration in TGI predates its adoption in vLLM and SGLang. When TGI launched it, Flash Attention was a significant competitive differentiator that reduced KV cache memory access and improved attention computation speed. As of 2026, Flash Attention is a baseline feature across all four frameworks. The integration advantage TGI had in 2022 is now a neutral baseline.
04KV configuration in TGI is intentionally simple: set the maximum GPU memory allocation and TGI manages the rest. This simplicity is appropriate for teams that want predictable, understandable behavior over tunable performance. The tradeoff is that workload-specific optimizations available in vLLM's scheduler and SGLang's radix tree are not accessible.
05The throughput cost of TGI's KV management approach versus vLLM and SGLang on prefix-heavy workloads is real. A multi-tenant RAG deployment where users share a consistent document corpus would see 30 to 80% lower throughput on TGI compared to SGLang with RadixAttention, purely from the difference in prefix caching efficiency. Teams running this workload pattern on TGI are funding the migration to SGLang indirectly through higher GPU bills.

TGI's KV cache is adequate and stable. It represents what good KV management looked like before PagedAttention and RadixAttention. Running it in 2026 for stable workloads that do not require partial prefix sharing is a defensible operational choice. Running it for new RAG or conversational workloads where prefix sharing is the primary throughput multiplier is an avoidable performance penalty.

03

How do they handle batching and scheduling?

A GPU serves one batch at a time. How you compose that batch — which requests go in, in what order, at what point new requests join — determines how much of the hardware's capacity you actually use. The difference between naive static batching and modern continuous batching is approximately 10x throughput on identical hardware.

View

Python · Apache 2.0

vLLM

Continuous Batching, Chunked Prefill

Batch Stack

Continuous batching

Chunked prefill (interleaved)

Cache-priority scheduling (v1)

Preemption with swap or recompute

Tensor and pipeline parallelism

01Continuous batching changes the batch composition after every decode step rather than waiting for all sequences in a batch to finish before adding new ones. As soon as one sequence completes, the scheduler slots in a waiting request without pausing the rest of the batch. On workloads with variable sequence lengths — which is most production traffic — this keeps GPU utilization high throughout the serving period instead of dropping whenever short sequences finish early.
02Chunked prefill splits long input prompts into smaller chunks that interleave with the decode steps of other active sequences. Without chunked prefill, a single 32,000 token prompt occupies the entire GPU during its prefill phase while every other waiting request stalls. With chunked prefill, the long prompt processes in slices across many steps, each interleaved with decode steps of other requests. The requesting user sees higher TTFT for their long prompt in exchange for dramatically lower tail latency for all other users during the same period.
03The v1 scheduler introduced priority-based scheduling that weights requests with higher prefix cache hit rates above requests that would require full prefill computation. Under load, a request with an 80% cache hit rate gets scheduled before a request with a 0% hit rate, because the cached request consumes less compute per token returned. This makes the cache more valuable at exactly the moments when it matters most: when the system is under pressure.
04When GPU memory fills under sustained high load, vLLM preempts the lowest-priority active sequences by moving their KV pages to CPU RAM or discarding them. Preempted sequences resume from their last cached checkpoint when GPU memory becomes available. This graceful degradation maintains service for in-flight requests under overload rather than failing entire batches, which is the behavior that production systems require.
05Tensor parallelism distributes model weights across multiple GPUs within a single node by splitting attention heads and feedforward weight matrices across GPU devices. vLLM's tensor parallelism implementation is stable across NVIDIA and AMD hardware. Pipeline parallelism (assigning different transformer layers to different GPUs in sequence) is also available for models that exceed single-node memory capacity. Both modes compose with continuous batching and chunked prefill without requiring scheduling changes.

vLLM's batching system covers the full production requirement: continuous batching for GPU utilization, chunked prefill for fairness under variable prompt lengths, priority scheduling for cache efficiency, and graceful preemption under overload. No single component of this is unique to vLLM but the combination is well-tested, well-documented, and compositionally sound. Teams that want to understand what their serving system does under load will find vLLM's scheduler the most inspectable of the four.

Python · Apache 2.0

SGLang

Zero Overhead, Radix-Aware Scheduling

Batch Stack

Zero overhead CPU scheduler

Radix-aware batch assembly

Expert parallelism (MoE)

Disaggregated prefill-decode

Structured output batching

01The CPU scheduler and GPU execution thread run in lockstep without blocking each other. The scheduler prepares the next batch on CPU while the GPU executes the current one. When the GPU finishes, the next batch is already assembled and submits immediately with no scheduling gap. In earlier vLLM versions and in TGI, occasional scheduling stalls blocked GPU execution. SGLang eliminated this as a latency source by designing the CPU scheduling path to always complete before the GPU step does.
02Radix-aware batch assembly: when selecting which waiting requests to include in the next batch, SGLang's scheduler considers which requests share cached prefixes in the RadixAttention tree. Requests that share a common cached prefix are more valuable to batch together than requests with no overlap, because the shared computation cost is already paid and the GPU cycles go entirely to new token generation. This scheduling awareness is what produces SGLang's throughput advantage on mixed workloads even before any benchmark tuning.
03Expert parallelism for Mixture of Experts models: SGLang routes each token to its selected expert on the GPU where that expert's weights reside, coordinating load balancing across experts to prevent some being continuously overloaded while others sit idle. For MoE models that activate only a subset of parameters per token (DeepSeek, Mixtral, and similar), this routing-aware scheduling is the primary driver of throughput efficiency. vLLM's general-purpose scheduler does not have explicit expert-routing awareness at the same level.
04Disaggregated prefill-decode scheduling in SGLang 0.5.x routes compute-heavy prefill requests to a dedicated pool of nodes optimized for dense computation and memory-bound decode requests to a separate pool. The Dynamo orchestration layer handles routing and KV transfer between pools. For deployments processing inputs longer than 16,000 tokens, teams report 40 to 60% TTFT reduction with disaggregated mode compared to co-located prefill-decode on the same hardware.
05Structured output batching: when multiple concurrent requests require constrained generation (JSON schema, tool call format, regular expression), SGLang batches the constraint validation computation via xgrammar on GPU alongside the standard decode. Frameworks that run constraint validation on CPU between GPU steps process structured output requests sequentially at the constraint step. SGLang's GPU-native constraint batching means structured output requests at high concurrency do not create a CPU bottleneck.

SGLang's scheduling improvements over vLLM are concentrated in three specific scenarios: workloads with significant prefix sharing (radix-aware assembly), MoE models (expert parallelism), and structured output at scale (GPU-native constraint batching). For standard chat or completion workloads with no shared prefixes and no structured output requirements, the scheduling difference is small enough that other factors dominate the framework choice. SGLang's scheduler is more sophisticated; whether that sophistication applies to your specific workload determines whether it translates into a meaningful throughput advantage.

C++ / Python · Apache 2.0 · NVIDIA Only

TensorRT-LLM

In-Flight Batching with CUDA Graphs

Batch Stack

In-flight batching

CUDA graph capture

Fused multi-head attention

Per-batch-size compiled engines

Speculative decoding (EAGLE3)

01In-flight batching in TRT-LLM executes via CUDA graph capture rather than individual kernel launches. A CUDA graph records the entire sequence of GPU operations for a given batch configuration as a single replayable object. During serving, each decode step replays the captured graph rather than launching individual kernels. At high decode step rates, the reduction in kernel launch overhead — typically 0.1 to 0.2ms per step saved — accumulates into measurable latency reduction across a long generation sequence.
02Fused multi-head attention in TRT-LLM merges the attention computation and the KV cache update into a single CUDA kernel for the decode phase. In frameworks that execute these as separate kernel launches, the GPU makes two memory-bandwidth passes over the KV cache per decode step. The fused kernel makes one pass. On the decode path, where memory bandwidth is the primary bottleneck, halving the number of KV cache memory passes directly improves throughput per decode step.
03TRT-LLM can compile separate engines for different maximum batch sizes and select among them at runtime. An engine compiled for maximum batch size 8 is optimized differently (different CUDA tile sizes, different memory layout assumptions) than one compiled for batch size 256. Serving interactive chat traffic through the batch-8 engine and batch processing jobs through the batch-256 engine produces better utilization than a single general-purpose engine attempting to optimize for both simultaneously.
04EAGLE3 speculative decoding (TRT-LLM v1.2.1): a draft model proposes multiple tokens for each decode step; the target model verifies them in a single parallel forward pass. Accepted tokens advance the sequence by multiple positions per step, reducing the total number of decode steps required for a given output length. TRT-LLM's EAGLE3 implementation compiles both draft and target models into the same engine, allowing the verification pass to run in the same CUDA graph as the standard decode step.
05The compilation-per-configuration approach creates operational inflexibility that shows up in three common scenarios: model updates require rebuilding, quantization changes require rebuilding, and hardware configuration changes require rebuilding. Teams with active CI/CD pipelines for model updates who also want TRT-LLM performance typically use NIM for stable model versions and maintain a separate pipeline for evaluating new model candidates on vLLM before committing to a TRT-LLM build.

TRT-LLM's batching optimizations — CUDA graphs, fused attention, per-batch-size engines — are the result of taking the interpretation overhead out of the hot path entirely. The performance gains from these optimizations are real and consistent on NVIDIA hardware. They are also the reason TRT-LLM requires a dedicated infrastructure workflow separate from vLLM or SGLang deployments. Teams must decide whether the throughput difference justifies the operational complexity difference.

Rust · Apache 2.0 · Maintenance Mode

Text Generation Inference

Continuous Batching, No Chunked Prefill

Batch Stack

Continuous batching

No chunked prefill

No disaggregated serving

Waterfall mode (burst workloads)

Simpler scheduling, no priority

01TGI implements continuous batching: new requests join the active batch when space frees after a decode step, without waiting for the full batch to complete. This is the same foundational technique as vLLM and SGLang. The scheduling logic is simpler — no prefix overlap prioritization, no radix-aware assembly, no expert routing. Requests join the batch when space is available and leave when their sequence finishes. Predictable, understandable, not optimal.
02Chunked prefill is not available in TGI. A single request with a 32,000 token input occupies the GPU for its entire prefill phase while all other waiting requests queue behind it. This head of line blocking is visible as latency spikes in the queue during periods when any long-context request arrives. vLLM and SGLang both eliminate this issue through chunked prefill. TGI deployments with mixed short and long context requests exhibit this behavior as a consequence of the architecture.
03Waterfall mode (configurable): batch requests that arrive within a set window before processing rather than dispatching each request as soon as it arrives. For workloads with bursty arrival patterns where grouping requests into larger batches improves GPU utilization, waterfall mode is beneficial. For interactive applications where adding waiting time to individual requests degrades user experience, it is counterproductive. The mode is deployment-configurable rather than learned from traffic patterns.
04No disaggregated serving support in TGI. Prefill and decode run on the same GPU in the same serving process. For workloads at the scale where disaggregating these phases becomes necessary (very long context, very high concurrency), TGI is not an option regardless of other preferences. This is a capability that is not expected to arrive given maintenance mode status.
05TGI's simpler scheduling is not inherently wrong for all workloads. At moderate concurrency with consistent short-to-medium prompt lengths and no significant prefix sharing, the gap versus vLLM's more sophisticated scheduler is small. The head of line blocking from absent chunked prefill matters only when long-context requests appear in the queue. Teams with well-controlled input length distributions on TGI may see less of this issue than teams with open-ended user inputs.

TGI's batching is a snapshot of what good production LLM batching looked like before chunked prefill and radix-aware scheduling existed. It works. It scales to many production use cases. It leaves measurable throughput unrealized on workloads with variable prompt lengths or shared context. The migration to vLLM or SGLang is primarily motivated by this throughput gap and the head of line blocking issue, not by correctness concerns with TGI's continuous batching foundation.

04

How does each framework handle quantization?

Quantization reduces the numerical precision of model weights and KV cache tensors. The memory saved from that reduction translates directly into larger batch sizes, longer context windows, or fewer GPUs for the same throughput. At production scale, a well-chosen quantization configuration is often more valuable than a faster GPU.

View

Python · Apache 2.0

vLLM

No Build Step, Broadest Format Support

Quant Stack

FP8, INT8, INT4

AWQ, GPTQ, MXFP4

FP8 KV cache

Load at startup, no rebuild

Multi-vendor quant support

01vLLM supports FP8, INT8, INT4, AWQ, GPTQ, and MXFP4 weight quantization. Quantized model checkpoints load from the Hugging Face Hub or local paths at startup without a separate compilation or conversion step. Changing quantization means pointing to a different checkpoint, not rebuilding the serving infrastructure. This operational simplicity is the primary reason vLLM is the default recommendation even for teams that will eventually need TRT-LLM's peak performance.
02FP8 weight quantization on H100 hardware produces 1.5 to 2 times throughput improvement versus FP16 at the same batch size, with less than 1% performance degradation on standard benchmarks for models at 70 billion parameters and above. The throughput gain comes from two sources: higher tensor core utilization (H100's tensor cores run FP8 operations faster than FP16) and smaller weight tensors requiring less memory bandwidth per forward pass.
03FP8 KV cache quantization is now a production recommendation for all H100 deployments. Storing key and value tensors in FP8 rather than FP16 halves the KV memory footprint. Combined with FP8 weight quantization, total GPU memory consumption for a typical inference workload drops 60 to 70% versus full FP16. That freed memory goes directly into serving more concurrent sequences or longer context windows on the same hardware.
04MXFP4 support arrived via the hardware plugin system in 2026. MXFP4 uses microscaling where a small group of weight values shares a single scaling factor. This grouping preserves more numerical dynamic range than standard INT4 quantization at equivalent bit widths. For very memory-constrained deployments where INT4 quality loss is unacceptable, MXFP4 offers a middle position between FP8 memory efficiency and FP8 quality.
05The practical quantization decision in vLLM: FP8 weights and FP8 KV for any H100 or newer deployment as the baseline configuration. INT4 or AWQ for scenarios where the model barely fits in GPU memory at FP8 and serving must happen regardless. FP16 only for the rare case where quantization error is genuinely measurable and unacceptable for the specific task — long-form mathematical reasoning and code generation with strict output requirements are the typical cited examples.

vLLM's quantization story is the most operationally simple of the four. The startup-time checkpoint loading means quantization configuration is a deployment variable rather than a build variable. Teams iterating on model versions and quantization levels can test configurations in minutes. This velocity advantage during model evaluation often outweighs TRT-LLM's throughput advantage at the same precision level, especially for teams making frequent updates to their production model.

Python · Apache 2.0

SGLang

Quantization Meets RadixAttention

Quant Stack

FP8, INT4, AWQ, GPTQ

NVFP4 on Blackwell (v0.5.x)

FP8 KV in radix tree

Per-layer mixed precision

Production traffic calibration

01SGLang supports the same quantization formats as vLLM with the same startup-time loading approach. NVFP4 on Blackwell GPUs was added in SGLang 0.5.x, making SGLang and TRT-LLM the two frameworks with production support for 4-bit Blackwell-native quantization. At NVFP4, a 70 billion parameter model that previously required four H100s can serve from two B200s with comparable output quality on most tasks. The Blackwell hardware generation improvement and the precision reduction compound into a significant cost per million tokens reduction.
02FP8 KV cache inside the RadixAttention tree: cached prefix blocks are stored in FP8 format and reused by subsequent requests without dequantization. The FP8 quantization error introduced when a prefix was first computed is inherited by every request that later reuses the cached block. In practice this error is unmeasurable in user-facing output quality. Architecturally it is worth understanding: shared KV cache precision is a global decision that affects all consumers of a cached prefix, not a per-request decision.
03Per-layer mixed precision is a documented SGLang configuration for long-context tasks: attention layers retain FP8 while feedforward layers use INT4. Attention layers in long-context generation are more sensitive to quantization error because small errors in the attention weight distribution compound across thousands of attended tokens. Keeping attention at FP8 while reducing feedforward to INT4 preserves long-context quality at lower average memory consumption than uniform FP8 would require.
04Calibration on production traffic: when running quantization calibration, SGLang's documentation recommends using a representative sample of actual production requests rather than a generic calibration dataset. Quantization calibration finds scaling factors that minimize error for the observed input distribution. A model that mainly processes financial documents calibrated on financial text will produce less quantization error than one calibrated on Wikipedia. Teams with domain-specific workloads consistently report lower quantization error with domain-calibrated quantization.
05SGLang's structured generation constraints interact with quantization. Constrained decoding adjusts the probability distribution over the vocabulary at each step to enforce format requirements. Quantization introduces small errors into that probability distribution. SGLang tests quantized models against constraint-heavy evaluation suites before recommending quantization levels for structured output workloads, because a quantization level that is invisible in free-text generation can occasionally cause constraint violations that are visible in strict JSON schema enforcement.

SGLang's quantization is equivalent to vLLM's in most dimensions and adds NVFP4 for Blackwell deployments and per-layer precision mixing for long-context quality. The practical implication: teams choosing between vLLM and SGLang on quantization grounds alone will find both adequate for standard FP8 deployments. For Blackwell with NVFP4 or for long-context workloads with precision concerns, SGLang's additional capabilities are relevant. For everything else, quantization is not the deciding factor between the two.

C++ / Python · Apache 2.0 · NVIDIA Only

TensorRT-LLM

Compiled Precision, NVFP4 Leader

Quant Stack

FP8, FP4, NVFP4 (Blackwell)

Per-precision compiled engine

ModelOpt calibration pipeline

FP4 weights plus FP8 KV (default)

Double throughput on B200 vs H100

01Quantization decisions in TRT-LLM are build-time, not startup-time. Selecting FP8 versus FP4 versus NVFP4 means building a different compiled engine. Each engine is fully optimized at the kernel, memory layout, and CUDA tile level for its target precision. The optimization depth is greater than startup-time loading but requires rebuilding whenever the precision configuration changes. Teams with fixed production quantization configurations pay the build cost once. Teams still evaluating quantization levels pay it repeatedly.
02NVFP4 on Blackwell: TRT-LLM was the first serving framework to deliver production NVFP4 support on B200 GPUs. NVFP4 is a 4-bit float format using microscaling specifically designed for Blackwell's new tensor core architecture. NVIDIA's internal benchmarks for TRT-LLM on B200 with NVFP4 versus H100 with FP8 show more than double throughput. The gain reflects both the hardware generation improvement and the precision reduction's memory bandwidth effect, compounding together.
03FP4 weights combined with FP8 KV cache is the recommended default for B200 deployments in TRT-LLM documentation as of mid-2026. A 70 billion parameter model in FP16 occupies approximately 140GB of weight memory. In FP4, the same model fits under 20GB. Combined with FP8 KV cache, the remaining GPU memory budget on a B200 is almost entirely available for KV cache, enabling very long context or very large batch at a fraction of the hardware that FP16 would require.
04NVIDIA's ModelOpt quantization toolkit integrates directly with the TRT-LLM build pipeline. Calibration data flows into ModelOpt, which measures the per-layer quantization sensitivity of the model and selects scaling factors that minimize error. The calibrated checkpoint then compiles into a TRT-LLM engine. For teams with access to representative calibration data and engineering time to run the full pipeline, the quality-versus-compression tradeoff at any given bit width is as favorable as the state of the art allows.
05Multi-precision comparison is straightforward in TRT-LLM: build the same model at FP8, FP4, and NVFP4, run each against your quality benchmark suite, measure throughput on your production hardware, and select the highest compression that meets your quality threshold. Each comparison is a separate build, but the framework makes the comparison reproducible and the results directly comparable because the evaluation runs against compiled engines rather than interpreted inference paths with variable overhead.

TRT-LLM's quantization story is the highest-performance and the least flexible of the four. The compiled precision advantage on Blackwell with NVFP4 is real and measurable — teams serving stable high-volume models on B200 hardware will find TRT-LLM's quantization produces the lowest cost per million output tokens available. The flexibility cost — a rebuild per precision configuration — is acceptable when quantization is a deployment decision rather than an evaluation variable. The two requirements work against each other during the model evaluation phase and together once a stable production configuration is established.

Rust · Apache 2.0 · Maintenance Mode

Text Generation Inference

FP8 Ceiling, No NVFP4

Quant Stack

FP8, INT8, INT4, GPTQ, AWQ

bitsandbytes integration

FP8 ceiling (no NVFP4)

ExllamaV2 support

Stable, not evolving

01TGI supports FP8, INT8, INT4, GPTQ, and AWQ quantization. The implementation is production-stable and documented. FP8 KV cache support was added before the maintenance period began. As of maintenance mode, no new quantization formats are being added. Teams targeting Blackwell hardware with NVFP4 requirements, or AMD hardware with MXFP4 requirements, need to migrate to vLLM or SGLang before adopting those hardware generations.
02bitsandbytes integration in TGI loads INT4 and INT8 quantized HF Hub checkpoints without a conversion step. For teams already using bitsandbytes-quantized models in their HF workflow, TGI accepts those checkpoints directly. vLLM and SGLang also support bitsandbytes quantized models, so this is not an exclusive TGI capability but it is a parity feature that eases migration in either direction.
03FP8 is the practical ceiling for TGI quantization. On H100 hardware, FP8 produces competitive throughput improvement versus FP16. Teams running TGI on H100 with FP8 quantization are not significantly disadvantaged versus vLLM on the same hardware in quantization efficiency. The gap emerges when the team moves to B200 hardware or when new quantization formats (NVFP4, MXFP4) become standard — at that point TGI cannot follow.
04ExllamaV2 model support allows TGI to serve EXLM-format quantized checkpoints popular in the community inference ecosystem. This format predates NVFP4 and remains in use for teams that built their inference pipelines around it. TGI's support for this format predates vLLM's and remains a migration consideration for teams with existing ExllamaV2 model libraries.
05For current TGI deployments on H100 with FP8, the quantization story is complete. The incentive to migrate specifically for quantization improvements is low unless the team is planning a hardware migration to Blackwell or AMD GPU generations. Teams that are hardware-stable on H100 and workload-stable on current model architectures can continue running FP8 on TGI without a meaningful quantization disadvantage versus vLLM for the foreseeable future.

TGI's quantization support covers what was state of the art in 2023 and 2024 and remains functional for those precision levels. The gap versus the competition is future-looking: teams that intend to migrate to Blackwell hardware or to adopt NVFP4 for its substantial cost-per-token improvements need a framework that supports NVFP4. That requirement alone is a sufficient migration trigger for many teams, independent of all other performance considerations.

05

How do latency and throughput trade against each other?

Every serving configuration is a point on a latency-throughput curve. Moving toward higher throughput means serving more requests per second at the cost of higher latency per request. Moving toward lower latency means faster individual responses at the cost of lower total capacity. Where each framework sits on that curve, and whether it lets you move along it deliberately, determines whether it fits your production requirements.

View

Python · Apache 2.0

vLLM

Balanced, Predictable Degradation

Numbers (H100, FP8)

TTFT: 30-80ms (short inputs)

TTFT: 500-1500ms (32K inputs)

ITL: 5-15ms at batch under 64

~2000-3000 TPS at peak

Graceful overload behavior

01Time to first token (TTFT) on a single H100 with FP8 serving Llama 3.1 70B: 30 to 80ms for short inputs under 1,000 tokens at moderate concurrency. At 32,000 token inputs under the same conditions, TTFT reaches 500 to 1,500ms. Chunked prefill reduces the impact of long inputs on other queued requests but does not reduce the absolute TTFT for the long-input request itself — the request still needs its full prefill computed before the first output token can be generated.
02Inter-token latency (ITL) — time between consecutive output tokens — is 5 to 15ms at batch sizes under 64 on H100 with FP8. As batch size increases, ITL increases proportionally (each decode step serves more sequences, taking more time) but total throughput in tokens per second summed across all sequences increases superlinearly. The tradeoff: small batches respond faster to individual users but serve fewer users simultaneously per GPU.
03Peak throughput on a single H100 FP8 serving Llama 3.1 70B: approximately 2,000 to 3,000 output tokens per second at maximum batch size with optimal settings. This ceiling reflects the H100's memory bandwidth divided by the bytes transferred per decode step for that model architecture. FP8 quantization increases this ceiling proportionally to memory savings by reducing the bytes per decode step. Moving to FP4 would increase it further but requires additional quality tradeoff evaluation.
04Under sustained overload, vLLM degrades gracefully rather than failing. The scheduler applies preemption, moving lower-priority sequences to CPU swap while maintaining service for higher-priority ones. New arrivals queue without receiving errors. The degradation curve is predictable and well-documented, which makes capacity planning reliable: teams can model the throughput-to-latency relationship at different traffic levels and set infrastructure provisioning targets with confidence.
05vLLM sits at a balanced position on the latency-throughput curve compared to the alternatives: not the lowest latency (TRT-LLM holds that on NVIDIA) and not the highest throughput on prefix-heavy workloads (SGLang's RadixAttention wins there). For workloads without strong prefix sharing and without NVIDIA hardware commitment, vLLM's balanced position is the correct default. The performance ceiling you are not reaching with vLLM is only relevant if you can measure the gap in your production context.

The most important practical insight about vLLM's performance profile is the predictability. Teams running load tests on vLLM can characterize the latency-throughput curve at different concurrency levels and trust that the production system will behave similarly. Predictable performance degradation enables honest SLA commitments. Frameworks that produce unpredictable tail latency spikes under load make SLA commitments difficult regardless of their median performance numbers.

Python · Apache 2.0

SGLang

Cache-Multiplied Throughput

Numbers (H100, FP8)

2.5-3x throughput at 70% cache hit

Under 5% overhead on structured output

25x vs H200 on DeepSeek R1 (GB300)

40-60% TTFT reduction (disaggregated)

Equivalent to vLLM at 0% cache hit

01At 70% RadixAttention cache hit rate (typical in multi-turn conversational workloads and shared-document RAG systems), effective throughput is 2.5 to 3 times higher than at 0% cache hit rate on the same hardware. The GPU cycles saved from cache hits go directly into processing additional requests. The GPU does not become faster; each cache hit simply eliminates a block of GPU work that the request would otherwise have required. The throughput gain is real and it scales with cache hit rate.
02In the February 2026 LMSYS and NVIDIA collaboration benchmark, SGLang running DeepSeek R1 on the GB300 NVL72 rack-scale system achieved 25x higher performance versus running the same model on H200. This number is not a pure SGLang software improvement — it reflects both Blackwell hardware architecture gains and SGLang's optimizations for the GB300's memory bandwidth and NVLink interconnect. It is included here because it represents the performance ceiling available when combining the best available hardware with the serving framework most optimized for MoE models.
03Structured output latency overhead versus unconstrained generation: under 5% additional latency per token in SGLang's benchmarks. External library-based constraint enforcement (Outlines integrated with vLLM) adds CPU-side overhead between GPU decode steps and produces higher overhead especially under concurrency. For applications that need JSON output from thousands of concurrent users, the latency difference between GPU-native and CPU-side constraint enforcement is measurable and accumulates across each token of the constrained output.
04Disaggregated prefill-decode: for deployments processing input contexts longer than 16,000 tokens, routing prefill to compute-specialized nodes and decode to memory-bandwidth-specialized nodes reduces TTFT for those long-context requests by 40 to 60% compared to co-located serving. The trade is infrastructure complexity: two node pools instead of one, a routing layer, and KV transfer infrastructure. For workloads dominated by long-context inputs, this trade is favorable. For typical chat inputs under 4,000 tokens, disaggregation produces negligible benefit.
05On workloads with no prefix sharing — completely unique prompts on every single request — SGLang and vLLM produce approximately equivalent throughput and latency on equivalent hardware. The RadixAttention advantage does not apply when there is nothing to cache. Teams evaluating SGLang for workloads with unique per-request prompts should test with realistic traffic before expecting the benchmark performance numbers, which typically reflect cache-heavy scenarios.

SGLang's latency and throughput numbers are workload-conditional in a way that vLLM's and TRT-LLM's are not. If your workload has significant prefix sharing, SGLang produces dramatically better efficiency than alternatives. If it does not, the difference is small. Evaluating SGLang requires characterizing your traffic's prefix sharing ratio first. Teams that skip this step and benchmark on synthetic unique-prompt workloads will underestimate SGLang's production advantage for their actual traffic patterns.

C++ / Python · Apache 2.0 · NVIDIA Only

TensorRT-LLM

Lowest Latency on NVIDIA

Numbers (H100, FP8)

10-30% lower TTFT vs vLLM

10-25% higher throughput vs vLLM

2x+ throughput on B200 vs H100

EAGLE3 speculative: 2-3x decode

30-60 min rebuild for config changes

01On a single H100 with FP8 serving a stable model configuration, TRT-LLM achieves 10 to 30% lower TTFT and 10 to 25% higher throughput than vLLM in published comparisons as of June 2026. These are directional numbers from InferenceEngineering.tech and Yotta Labs independent analyses; actual production numbers vary with model architecture, input length distribution, and batch composition. The improvement is real and consistent but not transformative for teams that are not yet throughput-constrained on vLLM.
02On Blackwell B200 with NVFP4, TRT-LLM records the highest published throughput numbers as of mid-2026. NVIDIA's internal benchmarks show B200 with NVFP4 delivering more than double the throughput of H100 with FP8 on equivalent models. Half of this gain comes from the Blackwell hardware generation improvement; the other half from NVFP4's memory bandwidth reduction. Both compound with TRT-LLM's compiled kernel optimizations to produce numbers that no other framework matches on NVIDIA Blackwell hardware.
03EAGLE3 speculative decoding in TRT-LLM v1.2.1: a draft model proposes multiple tokens per step; the target model verifies all proposals in one forward pass. Accepted tokens advance the sequence by multiple positions per step. The effective decode speedup is typically 2 to 3 times faster in practice on code and structured text generation tasks where the draft model achieves high acceptance rates. TRT-LLM's compiled implementation of EAGLE3 runs draft and target verification in the same CUDA graph, eliminating inter-model scheduling overhead.
04The rebuild cost of configuration changes creates a response latency to production problems that is structurally different from vLLM and SGLang. A performance regression in production that vLLM or SGLang teams diagnose, test, and fix with a configuration change in under an hour may require 30 to 60 minutes of rebuild time in TRT-LLM before the fix can be tested against production traffic. Teams should factor this incident response time into their production readiness assessment of TRT-LLM, not just the serving performance numbers.
05The cost per million output tokens on B200 with NVFP4 and TRT-LLM is the lowest available in open-source serving infrastructure as of mid-2026 for NVIDIA hardware commitments. At a production scale where GPU server costs represent a significant budget line, a 20 to 30% cost-per-token reduction versus vLLM on the same hardware translates into hundreds of thousands of dollars in annual infrastructure savings. This financial argument is the most common reason engineering teams invest in the TRT-LLM build infrastructure despite the operational complexity.

TRT-LLM's performance advantage is real, consistent, and financially meaningful at scale. The framework makes the right architectural trade for a specific use case: maximum throughput on stable NVIDIA workloads where configuration changes are infrequent and where the build infrastructure investment is a one-time cost. Teams that match this profile should use TRT-LLM. Teams that iterate frequently on models, quantization levels, or serving configurations are paying a continuous rebuild cost that eventually outweighs the per-token performance advantage.

Rust · Apache 2.0 · Maintenance Mode

Text Generation Inference

20-40% Below Alternatives

Numbers (H100, FP8)

20-40% lower throughput vs vLLM

Head of line blocking on long inputs

Competitive TTFT at short inputs

Low concurrency advantage: negligible

Migration gain: 30-80% throughput

01TGI's throughput on standard workloads (Llama-family models, moderate context lengths, no significant prefix sharing) is 20 to 40% below vLLM and SGLang on equivalent hardware based on 2025 benchmark data. This gap reflects two structural differences: the absence of chunked prefill creating head of line blocking on variable-length inputs, and the simpler scheduling producing lower batch efficiency on mixed workloads. These are not implementation bugs — they are consequences of TGI's design decisions from before these techniques existed.
02TTFT in TGI is competitive with vLLM at short input lengths where chunked prefill's absence does not matter — short prompts complete prefill quickly and queue others minimally. The gap emerges at inputs above approximately 4,000 tokens where TGI's single-phase prefill creates visible latency spikes in the queue for waiting requests. For workloads with consistent short inputs, TGI's TTFT is acceptable. For workloads with open-ended user inputs that occasionally include very long contexts, the tail latency is materially higher than vLLM.
03At low concurrency — a single user, a research deployment, a development server — TGI's throughput disadvantages relative to vLLM are negligible. The optimizations in vLLM and SGLang produce their advantages under sustained concurrent load where batching efficiency, cache hit rates, and scheduling overhead compound over many requests. A researcher running occasional queries will not observe a meaningful difference between TGI and vLLM in their daily usage.
04Teams that have measured their TGI performance and found it adequate for their current traffic levels are not making an incorrect decision by staying on TGI. The migration decision should be driven by one of three triggers: hitting the throughput ceiling on current hardware and needing more capacity before adding hardware, planning a migration to Blackwell GPUs where TGI does not support NVFP4, or adopting a new model architecture that TGI does not support in maintenance mode.
05Documented migration performance gains from TGI to vLLM or SGLang: teams report 30 to 80% throughput improvement depending on their workload's prefix sharing ratio and input length distribution. The lower end (30%) represents workloads with unique prompts and short inputs where TGI's absence of prefix caching and chunked prefill has minimal impact. The upper end (80%) represents RAG workloads with shared document context where RadixAttention dramatically reduces recomputation.

TGI's latency and throughput profile is appropriate for its maintenance mode status: good enough for many production workloads, meaningfully suboptimal for workloads that have grown in complexity since TGI was designed. The 20 to 40% throughput gap versus vLLM is not negligible at production scale — it represents 20 to 40% more GPU capacity required to serve the same traffic. Whether that cost is justified by avoiding migration risk depends on each team's specific scale and growth trajectory.

06

How do you deploy each framework at scale?

A serving framework that performs well in a single-GPU test but requires three months of infrastructure work to reach production is not production-ready. Deployment story determines which teams can actually use a framework and under what conditions. These four frameworks have very different deployment requirements, community resources, and hardware constraints.

View

Python · Apache 2.0

vLLM

One URL to Switch, Any Hardware

Deploy Stack

OpenAI compatible API

AMD MI300X production-ready

Ray Serve autoscaling

Prometheus metrics built in

Dynamo disaggregated support

01OpenAI-compatible API server available from a single Python command: python -m vllm.entrypoints.openai.api_server --model meta-llama/Llama-3.1-70B-Instruct. Applications built against the OpenAI SDK switch to vLLM by changing one base URL. No client-side code changes required. This API compatibility is the single most important deployment feature in vLLM's adoption story — it reduces the migration surface to infrastructure rather than application code.
02AMD ROCm support on MI300X and MI325X is production-ready as of ROCm 6.3 and vLLM 0.23.0. The AMD MI300X provides 192GB of HBM3 memory per GPU versus the H100's 80GB. Models requiring 2 H100s in FP16 fit on a single MI300X, simplifying the serving architecture by removing tensor parallelism requirements for that model size. Teams with AMD GPU fleets, or teams building vendor-diverse infrastructure to avoid single-vendor hardware dependency, find vLLM as their only production-quality option in this comparison for AMD hardware.
03Ray Serve integration provides multi-replica autoscaling for vLLM deployments that exceed single-instance capacity. A Ray Serve deployment scales from one to N vLLM replicas based on queue depth or custom metrics, distributes incoming requests across replicas, and handles replica health monitoring and restart. Kubernetes operators from community contributors enable vLLM deployment in container-orchestrated infrastructure without custom orchestration scripts.
04Prometheus metrics emitted by default: throughput in tokens per second, TTFT percentiles, queue depth, active request count, KV cache utilization, and prefix cache hit rate. These metrics require no additional instrumentation code. Grafana dashboards for these metrics exist in the community repository. OpenTelemetry trace export is available for teams routing inference observability into existing monitoring infrastructure alongside application metrics.
05NVIDIA Dynamo integration makes vLLM a supported decode backend in disaggregated serving configurations. Dynamo routes prefill-heavy requests to prefill-specialized vLLM nodes and decode requests to decode-specialized vLLM nodes, handling KV page transfers between pools over NVLink or high-bandwidth Ethernet. For clusters serving millions of daily requests where disaggregation becomes the right infrastructure decision, vLLM participates in that architecture without migration to a different serving framework.

vLLM's deployment story is the clearest default recommendation in this comparison. The one-command startup, OpenAI API compatibility, hardware breadth, built-in Prometheus metrics, and community ecosystem of Kubernetes operators and Ray Serve integrations make the path from development to production faster than any alternative. The deployment advantage compounds with the model support breadth advantage: new models are supported faster in vLLM than alternatives, and deployment of those models requires less infrastructure work than alternatives.

Python · Apache 2.0

SGLang

PyTorch Native, MoE Deployment Leader

Deploy Stack

OpenAI compatible API

PyTorch ecosystem (March 2025)

MoE specialist deployment

Structured output endpoint

NVIDIA Dynamo supported

01OpenAI-compatible API server available identically to vLLM. The server startup flags differ slightly but the client-side experience is identical. Teams benchmarking SGLang against vLLM can test with a base URL change on an existing application, observe performance differences under real traffic patterns, and make the migration decision without building a separate test environment. This makes A/B evaluation between vLLM and SGLang operationally straightforward for any team that currently runs vLLM.
02PyTorch native integration (March 2025) means enterprises standardized on PyTorch infrastructure treat SGLang as part of the core ML stack rather than as an external dependency to evaluate and approve separately. For organizations where ML infrastructure decisions go through a PyTorch-centric approval process, the PyTorch official support status of SGLang substantially reduces adoption friction compared to pre-2025.
03MoE model deployment is where SGLang's production deployment differentiation from vLLM is clearest. For DeepSeek, Mixtral, and similar MoE architectures, SGLang's expert parallelism and routing-aware scheduling produces higher throughput on equivalent hardware than vLLM's general-purpose scheduler. Teams running MoE models at scale should benchmark both, but SGLang is the expected winner and the framework more likely to receive MoE-specific optimization investment going forward.
04Dedicated structured output endpoint: SGLang supports exposing a separate serving endpoint with different timeout, maximum batch size, and queue configuration specifically for constrained generation requests. Teams with mixed traffic (some requests needing JSON output, others needing free text) can route request types to the appropriate endpoint from a single SGLang deployment without running separate serving processes for each traffic type.
05Community size relative to vLLM: approximately 20,000 GitHub stars versus vLLM's 50,000+ as of mid-2026. This gap reflects vLLM's earlier launch and broader initial marketing, not a quality difference in active development. SGLang receives active maintainer responses to issues, weekly releases, and rapid new model support. The smaller community means fewer community-contributed Kubernetes operators and deployment tooling — teams deploying SGLang at scale typically build their own orchestration layer rather than relying on community templates.

SGLang's deployment story is where it is most similar to vLLM and where the decision between the two is most workload-dependent. The API compatibility, PyTorch integration, and NVIDIA Dynamo support mean that switching between vLLM and SGLang is genuinely low-risk. The deployment investment in either framework is largely transferable to the other. This means teams can start with vLLM, characterize their traffic's prefix sharing ratio in production, and migrate to SGLang later if RadixAttention proves beneficial for their actual workload — with minimal sunk cost.

C++ / Python · Apache 2.0 · NVIDIA Only

TensorRT-LLM

Build, NIM, Triton, Enterprise

Deploy Stack

Build then serve workflow

NIM (zero build, supported models)

Triton Inference Server

Dynamo primary backend

Confidential computing (H100/B200)

01The standard TRT-LLM deployment workflow: build an optimized engine per model-hardware-precision combination using NVIDIA's Docker containers, then deploy the compiled engine with the TRT-LLM runtime. Docker images from NVIDIA's container registry are preconfigured for H100 and B200 hardware. The build step takes 30 to 60 minutes for 70 billion parameter models and can run on a separate build node before the engine is transferred to serving infrastructure. One engine can be built once and deployed to multiple serving replicas without per-replica rebuilds.
02NIM (NVIDIA Inference Microservices) removes the build step for supported models. Prebuilt, preoptimized TRT-LLM containers for Llama, Mistral, Nemotron, DeepSeek, and other major architectures deploy with a single Docker command and an NVIDIA AI Enterprise license key. NIM containers include the serving API, health endpoints, Prometheus metrics, and model configuration optimized for the specified hardware without requiring build toolchain knowledge. For supported models on NVIDIA hardware, NIM is the fastest path to production quality TRT-LLM performance.
03NVIDIA Triton Inference Server integration deploys TRT-LLM engines as Triton backends. Triton provides enterprise-grade multi-model serving with gRPC and REST endpoints, model version management, concurrent model execution, and GPU resource allocation. Organizations already using Triton for computer vision or classical ML models add LLM serving to the existing Triton deployment without new infrastructure. The unified Triton management layer covers all model types simultaneously.
04NVIDIA Dynamo is designed around TRT-LLM as its primary compute backend for disaggregated cluster-scale serving, though vLLM and SGLang are also supported. Dynamo handles KV-aware routing between prefill and decode node pools, dynamic load balancing based on KV cache state, and infrastructure health management at cluster scale. For organizations serving millions of concurrent users across many GPUs, Dynamo with TRT-LLM is the highest-performance configuration available from any combination of open serving tools.
05Confidential computing on H100 and B200: TRT-LLM executes model weights and inference computation inside a hardware Trusted Execution Environment. The cloud provider's operators cannot access model weights or inference inputs in plaintext during execution. This hardware-level guarantee is relevant for organizations with strict model IP protection requirements — pharmaceutical companies, financial institutions, and government agencies whose model weights represent proprietary intellectual property that must remain confidential even from their cloud infrastructure provider.

TRT-LLM's deployment story is the most enterprise-structured of the four: NIM for rapid deployment of supported models, Triton for integration into existing ML serving infrastructure, Dynamo for cluster-scale orchestration, and confidential computing for organizations with IP protection requirements. This structure requires NVIDIA hardware commitment and engineering investment in the build toolchain. For organizations that have made or intend to make that commitment, the deployment story provides capabilities (model IP protection, Dynamo orchestration, per-precision compiled engines) that the other frameworks cannot match.

Rust · Apache 2.0 · Maintenance Mode

Text Generation Inference

Docker-First, HF Native, Migrate for New Projects

Deploy Stack

Single Docker command

HF Inference Endpoints

OTel + Prometheus built in

One URL migration to vLLM

No Blackwell NVFP4 support

01Docker-first deployment: docker run ghcr.io/huggingface/text-generation-inference --model-id meta-llama/Llama-3.1-70B-Instruct. No Python environment setup, no build steps, no configuration files beyond the Docker command flags. For teams whose infrastructure is Docker-based and who want the lowest setup friction of the four frameworks, TGI's Docker story achieves exactly this. The model downloads from the HF Hub automatically on first startup.
02HF Inference Endpoints provides managed TGI deployment on AWS, Azure, or GCP through the Hugging Face Hub. Select a model, select hardware, and a TGI instance is running within minutes. For teams that want managed LLM serving without building infrastructure, Inference Endpoints provides this without evaluating third-party managed vLLM or SGLang providers. The managed service handles scaling, availability, and model updates within the HF ecosystem.
03OpenTelemetry and Prometheus metrics built into TGI without additional configuration. Self-hosted TGI deployments emit standard Prometheus metrics that integrate with any existing Prometheus and Grafana monitoring setup. HF Inference Endpoints provides a hosted metrics dashboard. For teams evaluating observability setup costs across frameworks, TGI matches vLLM's built-in observability coverage without requiring additional tooling.
04Migration path from TGI to vLLM: the client-side migration is a base URL change. Teams running TGI behind a load balancer can add vLLM instances and gradually shift traffic from TGI to vLLM, validating output quality and latency under real production load before decommissioning TGI instances. The migration risk is low and the operational gain is typically 30 to 80% throughput improvement without hardware changes. For teams at their TGI throughput ceiling, this migration is the most available capacity upgrade before additional infrastructure spend.
05Blackwell GPU support: TGI does not support NVFP4 quantization and does not have documented Blackwell-specific optimizations in maintenance mode. Teams planning hardware migrations from H100 to B200 or GB200 should complete the TGI-to-vLLM migration before the hardware migration. Running TGI on Blackwell hardware foregoes the NVFP4 throughput gains that make Blackwell investments financially justified. The two migrations are independent but the hardware migration motivates the framework migration.

TGI's deployment story is the simplest of the four for teams starting from zero with HF infrastructure. The Docker command, the managed Inference Endpoints service, and the HF Hub integration make it the fastest path from a model checkpoint to a running inference endpoint for teams inside the HF ecosystem. Maintenance mode does not change this deployment advantage. It simply means the deployment ceiling — in terms of performance optimizations, hardware support, and new features — is fixed at its current state while the alternatives continue to grow.

M

Methodology

What counts as fact, what counts as inference, and what to verify before making infrastructure decisions based on this research.

📄

Official Documentation

Framework docs, GitHub READMEs, release notes, and changelogs for vLLM 0.23.0, SGLang 0.5.13, TRT-LLM 1.2.1, and TGI. Verified at research time, July 2026. Version numbers are accurate as of research date.

📊

Published Benchmarks

InferenceEngineering.tech (June 2026), Yotta Labs (2026 inference comparison), Spheron ROCm vs CUDA analysis (April 2026), GPUAdvisor MI300X guide (April 2026), and LMSYS-NVIDIA February 2026 SGLang-GB300 collaboration results. All directional — hardware and workload vary.

⚠️

Maintenance Mode Status

TGI maintenance mode (late 2025) is documented from the official HF recommendation to migrate new projects to vLLM or SGLang, confirmed across multiple independent 2026 reviews. The recommendation predates this research.

💭

Author Synthesis

Architectural comparisons, tradeoff assessments, and workload recommendations are the author's interpretation of primary sources. They appear under the insight label. Disagree with the reasoning, not the source claim.

On benchmark numbers

Every performance number in this study is directional. LLM inference performance varies significantly with model architecture, input length distribution, batch composition, GPU model, quantization configuration, and concurrent request count. Numbers from benchmarks using Llama 3.1 70B on H100 with FP8 at moderate concurrency — the most common benchmark configuration — do not predict performance for a 7B model on A100 with INT4 at high concurrency. Run benchmarks against your specific workload on your specific hardware before making infrastructure commitments. The relative relationships between frameworks (which is faster for which workload type) are more reliable than absolute numbers.

What is excluded

LMDeploy (Alibaba, strong for Qwen and InternLM models), llama.cpp and Ollama (CPU and edge focus, not production GPU serving), MLC-LLM (cross-platform browser and mobile), and commercial closed-source inference providers (Together AI, Fireworks, Baseten) were excluded. LMDeploy is worth evaluating for teams deploying Chinese-language models or on Ascend NPU hardware. llama.cpp is the correct choice for edge and on-device deployment. These are valid tools outside the scope of this volume's focus on production GPU cluster serving.

Research timeline

Researched July 2026. Primary sources were official framework documentation and GitHub repositories as of July 2026. Secondary sources included InferenceEngineering.tech's June 2026 vLLM vs SGLang vs TRT-LLM comparison, Spheron's April 2026 ROCm vs CUDA production analysis, GPUAdvisor's April 2026 AMD MI300X guide, Yotta Labs' 2026 inference engine comparison, GIGAGPU's May 2026 vLLM on ROCm guide, and the February 2026 LMSYS-NVIDIA SGLang GB300 benchmark announcement. Pasted research documents provided by the author supplemented official source verification.

Last Updated: July 24, 2026

Swarnim
Tiwari

AI Systems Researcher

There is a pattern in this series I did not plan but that keeps appearing. The tool that was first in a category, the one everyone trusted, ends up in maintenance mode by the time I write the volume about it. TGI here. Helicone in Volume 04.

The reason is always the same. The infrastructure underneath gets more complex than the tool was designed to handle. A faster team builds something new that matches the new requirements. The old tool freezes where it is.

Inference infrastructure is where the economics of AI actually live. The gap between a naive model runner and an optimized serving framework is not 10%. PagedAttention showed it was closer to a factor of 10 in concurrent sequences on the same GPU. That is the difference between a demo and a product that serves real users.

I am a student in India. This volume took longer to understand than the others. The right order to learn this material is: understand what prefill and decode actually mean, then understand why KV cache is the bottleneck, then understand what continuous batching is solving, and only then do the framework comparisons make sense.

LinkedIn GitHub

AI Systems Studies — Publication Series

Vol. 01Production AI Architecture — OpenAI, Anthropic, Palantir, NVIDIAPublished

Vol. 02AI Agent Frameworks — OpenAI SDK, LangGraph, CrewAI, MastraPublished

Vol. 03Vector Databases — Pinecone, Weaviate, Milvus, QdrantPublished

Vol. 04AI Observability — LangSmith, Langfuse, Helicone, W&B WeavePublished

Vol. 05Inference Infrastructure — vLLM, SGLang, TensorRT-LLM, TGIThis Study

Vol. 06Context EngineeringPlanned

Vol. 07Memory SystemsPlanned

Vol. 08RAG ArchitecturesPlanned