vLLM or SGLang, which should you use? Comparison posts disagree, so I added my own ARC-AGI competition measurements and sorted it out by GPU memory budget

About this post
Co-edited with Claude (Anthropic).

Summary

The general-purpose default is vLLM. It supports the widest range of models and hardware, and the major orchestration layers use it as a backend

SGLang is strongest on workloads that reuse prefixes (agents, multi-turn, structured output) and on multi-node serving of large MoE models such as the DeepSeek family

Performance comparisons flip with the workload. In a 2026 paper, vLLM was faster on dense models and SGLang was faster on MoE. Most public benchmarks don’t measure conditions with heavy prefix sharing

In ARC-AGI-2, vLLM won on direct answers (418 vs 272 tok/s) and multi-turn showed no difference. In ARC-AGI-3, moving to an SGLang-based stack raised the public LB from 2.68 to 11.07 (the contributing factors aren’t separated)

My reading is that the cases where the gap opened can be explained by whether the KV cache fits on the GPU. SGLang’s prefix reuse is strong, but on hybrid models it pays for saved state with memory

My conclusion: when in doubt, start with vLLM, but check the KV budget before the engine’s reputation. Choose the setup where concurrency × context length fits

How this started#

In autumn 2026 I was competing in Kaggle’s ARC Prize 2026 (ARC-AGI-2 and ARC-AGI-3)¹. Both are competitions where you bring weights into an offline Kaggle notebook and run inference within a limited time, so the tokens you can generate per second feed directly into your score. I ended up having to think about serving as much as about the model itself.

But when I looked things up, the conclusions didn’t line up. Some posts say SGLang is faster than vLLM, others say the opposite. When I swapped engines myself, ARC-AGI-2 and ARC-AGI-3 gave opposite results too.

So I went back through official sources and papers to sort out which one is strong in which situation, and I’ve added my measurements from swapping engines under the same load in the competitions. Versions and star counts from official sources were checked on 2026-11-04.

Here’s the answer up front: rather than asking which is faster, first check which one lets your workload fit in GPU memory.

The problem serving frameworks are solving#

The bottleneck in LLM inference is how the KV cache (the Keys / Values of past tokens) is handled in GPU memory. The differences between frameworks show up mainly in three areas.

Area	What it means	Representative techniques
Memory management	Pack KV without waste and raise the number of requests handled at once	PagedAttention (vLLM)³
Reuse	Reuse the KV of identical prefixes instead of recomputing it	RadixAttention (SGLang)⁷, prefix caching (vLLM)¹²
Scheduling and distribution	How batches are formed, separating prefill and decode, spreading across GPUs	continuous batching, PD disaggregation, Expert Parallel

If you use Claude Code, think of reuse as the same idea as prompt caching. In Anthropic’s API, only the part that matches exactly from the start, in the order tools, system, messages, hits the cache, and changing something midway invalidates everything after it². A coding agent harness (Japanese), with the system prompt and CLAUDE.md pinned at the start and history growing behind them, is the workload where prefix reuse pays off most.

Comparing vLLM and SGLang#

Aspect	vLLM	SGLang
Design starting point	Memory efficiency through PagedAttention³	Prefix reuse through RadixAttention⁷
How it describes itself	Fast, cheap LLM serving for everyone⁴	Optimized for agents, large-scale serving, and RL rollouts⁶
Governance	PyTorch Foundation⁵	LMSYS + the commercial company RadixArk⁶¹¹
Latest version	v0.32.0 (2026-10-18)⁸, with v0.33.0 at RC	v0.5.23 (2026-11-02)¹⁰
GitHub stars	About 96k	About 38k
Supported models	200+ architectures⁴	Major new models added with each release¹⁰. vLLM’s coverage is broader
Supported hardware	NVIDIA, AMD, Intel, plus TPU, Gaudi, Spyre, Ascend, Apple Silicon, and more⁴	NVIDIA, AMD, TPU, Intel, Apple Silicon, Ascend, Moore Threads⁶
Prefix reuse	Matched with a hash chain of fixed-length blocks. Only full blocks are cached¹²	Matching parts reused automatically with a radix tree. Routing is cache-aware too⁹
Beyond text generation	Mostly text output	Ships SGLang Diffusion for image and video generation⁶

Both release roughly every two weeks, and a feature that only one of them has often lands in the other within a few months. Treat comparison posts from six months ago as rough reference only.

Where vLLM is strong#

General-purpose production API servers: new models get supported quickly, and most models just run⁴
Non-NVIDIA hardware: if you’re on TPU, Gaudi, Ascend, and so on, it’s the first candidate⁴
Compute efficiency on dense models: in CCL-Bench (below), its MFU was about twice SGLang’s¹³
Integration with surrounding tools: llm-d, AIBrix, Ray Serve LLM, and Dynamo all support it²³²⁴²⁵²⁶

Where SGLang is strong#

Workloads with heavy prefix sharing: the paper reports up to 6.4× the throughput of then-existing systems on agent control, JSON decoding, multi-turn chat, and so on⁷. There’s also a case where a cache-aware load balancer raised the hit rate from 20% to 75%⁹
Structured output: compressed FSMs and xgrammar speed up constrained decoding such as JSON⁷⁹
Multi-node serving of large MoE models: with PD disaggregation and large-scale Expert Parallel, there’s a case of serving DeepSeek-V3 on 96 H100s¹⁴
RL rollout generation: it’s built into verl, slime, AReaL, and others⁶

What the benchmarks say#

Performance comparisons flip with conditions. Here are two whose conditions are relatively clear.

CCL-Bench 1.0 (2026-05, arXiv) compared them on A100s with the communication backend and parallelization settings aligned¹³.

Workload	Result	The paper’s analysis
Qwen3-4B (dense)	vLLM is faster. MFU 17.1% vs 8.6%	vLLM’s compute kernels are more efficient
Llama-3.1-8B (dense)	vLLM is faster. MFU 28.4% vs 15.5%	Same as above
DeepSeek-MoE-16B (EP4 / TP4)	SGLang is faster. 76.1 ms vs 81.2 ms	SGLang’s AllToAll implementation is more efficient

Jarvislabs’ blog post (2026-05, H100) found that vLLM had the best balance on short prompts, and that SGLang had the lowest TPOT on long 16K-token prompts. However, its TTFT was on the high side, versions weren’t recorded, and the author cautions that this isn’t a universal ranking¹⁵.

Neither one measures conditions with shared prefixes. That’s where SGLang really shines, so for agent use these numbers can’t settle it. Plenty of other blogs post numbers like “SGLang is 29% faster,” but their conditions are unclear and they contradict each other, so I didn’t use them here.

The way you measure also shifts the numbers. I covered what to watch for when comparing yourself in my post on load-testing tools (Japanese).

What happened when I swapped engines in ARC-AGI#

From here on, these are my own experiment notes. Both ran inference with Qwen-family models on Kaggle. All LB values are public scores; private scores haven’t been announced yet.

ARC-AGI-2: NVIDIA L4 (24 GB) × 4, connected over PCIe, Qwen3.8-27B (a mix of dense and GDN)
- Direct answer: think at length on a single prompt, then answer. Context grows to around 80k tokens, and prefixes aren’t reused
- Multi-turn: solve while calling a Python tool
ARC-AGI-3: RTX PRO 6000 (96 GB) × 1, Qwen3.8-Flash-Next (a 125B MoE). An agent that operates a game, with images and history growing over time

Workload	Setup	Concurrency	KV capacity (tokens)	Result
ARC2 direct	vLLM plain	24	1,408,000	298.5 tok/s
ARC2 direct	vLLM + MTP 2	21	1,157,090	418.1 tok/s (adopted)
ARC2 direct	SGLang + MTP	24 (actually 9–12)	516,014	272.1 tok/s
ARC2 direct	SGLang plain (no radix)	24	1,307,584	OOM on the first prefill
ARC2 multi-turn	vLLM	24	1,436,259	LB 16.11 (1 run)
ARC2 multi-turn	SGLang	24	752,964	LB 16.25 (mean of 2 runs)
ARC3	vLLM (KV fixed at 5 GiB, MTP 3)	28 × 32k	5 GiB	LB 2.68 (mean of 9 runs)
ARC3	vLLM (no MTP)	7 × 64k	486,838	LB 4.46 (mean of 2 runs)
ARC3	SGLang-based fork (FP8 KV, MTP 3)	10 × 128k	1,011,264	LB 11.07 (1 run)

MTP is speculative decoding bundled with the model; it uses extra KV in exchange for speed. The ARC-AGI-2 versions were vLLM 0.30.0 and SGLang 0.5.20, and the ARC-AGI-3 fork is equivalent to 0.5.19.

I learned four things.

First, SGLang’s loss on direct answers wasn’t because the engine was slow. SGLang + MTP lost 3.52 GB per GPU to GDN’s intermediate state, which cut KV to 516k, and only 9–12 of the 24 concurrent requests actually ran. The plain setup hit OOM in the prefill CUDA graph. In other words, this was the result of not fitting on a 24 GB GPU, not a comparison of engine speed. In another author’s post on RTX 4090 × 4, single-request decode was about 1.35× faster on SGLang, while KV was 281k vs 441k¹⁶. The direction matches.

Second, on multi-turn the reuse difference showed up, but the score didn’t move. Comparing the same window, prefix reuse was 0.955 for SGLang vs 0.723 for vLLM, and new prefill per turn was 702 vs 4,164 tokens (one run each). Decode speed was the same in the steady-state window. Meanwhile, KV was about half.

The reason lies in how hybrid models are built. Linear attention layers like GDN hold a fixed-size state rather than per-token KV. You can’t remove just the tokens after some midpoint, so reuse requires saving the intermediate state. SGLang’s extra_buffer holds 5 slots of state per request. Reuse isn’t free; you buy it by giving up KV. The LB swings between 15.00 and 17.50 even on the same version, so at that resolution I read this as no difference.

Third, in ARC-AGI-3 the score rose after switching to a setup with about twice the KV. In the vLLM image at the time, this MoE’s KV was BF16 only and couldn’t be made FP8. An SGLang-based fork could use FP8 KV and MTP 3 at the same time, which made 10 requests × 128k realistic. I took this straight from the serving stack in Daniel Franzen’s public solution¹⁷. The cache hit rate rose from 48% to 84% (measured under different conditions).

However, this 11.07 is the result of changing the weights, KV type, MTP, window, and concurrency all at once, so it can’t be separated out as an engine difference. It also wasn’t plain SGLang but a patched fork called Pennyroyal¹⁸. Switching to upstream 0.5.21 was estimated to reduce KV to around 820k, so I held off on that. The public solution itself scores an LB of 27.24, and I read the gap up to that as coming from parts other than serving.

Fourth, the penalty when KV fills up is large. In an ARC-AGI-2 production run, generation dropped from 374 to 218 tok/s, about 40%, in the stretch where KV was the limiting factor. That’s because decode stalls for the prefill of waiting requests and for preemption. In ARC-AGI-3 too, just going from 10 to 12 concurrent requests took the hit rate from 0.94 to 0.82 and cut the tokens generated on the game side by 18%.

OpenAI-compatible doesn’t mean swapping gives you the same input#

Here are the traps I hit while swapping. Both have OpenAI-compatible APIs, so swapping looks easy, but prompt rendering changes.

SGLang handles tools differently, which changed the system JSON by 11 tokens. The first attempt lost 20.17 to 24.00, but a second attempt with this rendering, the suspected cause, aligned tied at 24.00
SGLang’s Qwen3.8 template only reads the assistant’s reasoning_content, so if you send reasoning, all past reasoning is silently dropped

Before comparing, you need to confirm that the same input ends up as the same token sequence.

How to read the KV budget#

Here’s everything so far turned into steps you can apply to your own environment. The KV you need can be roughly estimated like this:

1
KV needed ≈ concurrency × (context length actually reached + fixed per-request overhead)

The fixed per-request overhead for Qwen3.8-27B on vLLM was equivalent to 1.5k tokens plain, 4.8k with MTP 2, and 7.4–8.6k with MTP 3. For context length, you can estimate from the realistic average rather than the maximum. The public ARC-AGI-3 solution has only 1.01M of KV against 10 requests × 128k (1.31M), but it holds the average to 88k by discarding down to 58k in one go when a request reaches 128k. Discarding a little at a time would break the prefix every turn, so it discards all at once. The idea is close to Claude Code’s /compact.

Here’s what consumes KV and where to check it.

Item	Effect on memory	Where to check
Weight quantization	Lighter weights leave more for KV	Memory breakdown in the startup log
KV type (BF16 / FP8 / NVFP4)	FP8 is about half of BF16	`--kv-cache-dtype`. Availability varies by model and version
Speculative decoding	Adds draft state and fixed per-request overhead	Compare across MTP levels
Hybrid state saving	Reserves slots for reuse	SGLang’s `--mamba-radix-cache-strategy`
CUDA graph	Eats the margin left after KV is allocated	SGLang’s `--mem-fraction-static`

After startup, look at these values.

vLLM: the log line Maximum concurrency for 81,920 tokens: 4.69x. If this multiplier is below your concurrency, it won’t fit. Count whether things got stuck with vllm:num_preemptions_total
SGLang: max_total_num_tokens in the startup log and full token usage on Decode lines. --enable-cache-report gives you cached_tokens per turn

The assumptions behind the budget keep changing too. vLLM 0.32 made NVFP4 KV work on SM8x and SM12x as well⁸. On my L4s, capacity went up 1.62× while tok/s went to 0.969×, so I didn’t adopt it for this workload. SGLang 0.5.23 added Paged Experts, which runs MoE models that don’t fit in GPU memory¹⁰. The numbers in the table above can inform how you choose a version, but they don’t rank today’s versions.

Other options#

The other options are easier to sort out by “layer.”

Inference engines (run on their own)#

Framework	Positioning	Good for
TensorRT-LLM	NVIDIA GPUs only. Custom kernels, PD disaggregation, speculative decoding¹⁹	Production environments squeezing maximum performance out of NVIDIA GPUs
LMDeploy	Two engines: C++ TurboMind and PyTorch²⁷	Quantized models, VLMs such as InternVL, Ascend
llama.cpp	Written in C/C++. Runs on CPUs and Apple Silicon with GGUF²⁸	Local, edge, small GPUs
Ollama	Based on ggml. Fetching and launching models is easy²⁹	Trying things out and development on a personal PC
MLC-LLM	Builds for each environment with an ML compiler³⁰	Phones, browsers
KTransformers	Puts frequently used experts on the GPU and the rest on the CPU²²	Running huge MoE models on few GPUs
TGI	Made by Hugging Face. Archived²¹	Avoid for new projects. HF itself recommends vLLM, SGLang, and others

TensorRT-LLM is often described as needing models compiled in advance, but that’s out of date. In 1.2 the TensorRT backend and trtllm-build were removed, and PyTorch is now the execution backend²⁰. When I compared small models against vLLM on one L4, windowed decode was about 12% faster for 9B and about 21% faster for 4B, but after re-aligning how I measured, the result varied with concurrency (9B: 1.08× at 21 requests, 0.96× at 6).

Orchestration layer (bundles engines from above)#

On a single node, the engine alone is enough. You add this layer when you need KV-aware routing, PD disaggregation, and autoscaling across multiple nodes. It doesn’t replace vLLM or SGLang; it uses them as backends.

Framework	Developer	Supported engines	Characteristics
NVIDIA Dynamo²³	NVIDIA	SGLang, TensorRT-LLM, vLLM	Offloads KV from GPU to CPU to SSD to remote, SLA-based autoscaling
llm-d²⁴	Red Hat, Google Cloud, IBM Research, and others. CNCF Sandbox	Centered on vLLM (also supports SGLang’s PD disaggregation)	Kubernetes native. Tiered KV offload
AIBrix²⁵	vllm-project (originated at ByteDance)	vLLM, SGLang, TensorRT-LLM	Routing that looks at engine state, dynamic LoRA loading
Ray Serve LLM²⁶	Anyscale	vLLM, SGLang	Multi-node serving and batch inference on Ray

Rule-of-thumb guide#

If this is your case	First candidate	Notes
Just stand up an OSS model behind an API	vLLM	When in doubt, this one
Using non-NVIDIA hardware	vLLM	SGLang supports some too. Check first
A just-released model or an unusual architecture	vLLM	Check support in both
Agents, multi-turn, long shared prompts	Measure SGLang	On hybrid models, subtract the memory for saved state when comparing
Producing lots of structured output such as JSON	SGLang
Long-form inference with no prefix reuse	Measure vLLM	The radix advantage disappears; it comes down to decode and KV
Serving a huge MoE across multiple nodes	SGLang	vLLM support is advancing too. Measure both
Maximizing concurrency on a small-VRAM GPU (such as 24 GB)	Compare KV capacity first	Subtract the CUDA graph and speculative state too
RL rollout generation	Depends on your RL framework
Local PC, edge	llama.cpp / Ollama	MLC-LLM for phones and browsers
Pushing NVIDIA to its limit	TensorRT-LLM	Combine with Dynamo
Large-scale Kubernetes operation	Either engine	Put llm-d / Dynamo / AIBrix on top

My conclusion#

When in doubt, start with vLLM, but check the KV budget before the engine’s reputation. Use the startup logs to compare which setup lets concurrency × context length fit on your GPU.

In ARC-AGI-2, SGLang was at a disadvantage because its KV was cut in half, and in ARC-AGI-3 the SGLang-based stack was at an advantage because it could use FP8 KV. I read these two seemingly opposite results as explainable on the same axis. The general claim that “SGLang is strong for agents” isn’t wrong, but if you choose on that alone, you’ll miss the memory you pay for reuse.

Finally, some caveats. My measurements cover a limited set of models and GPUs, and each is about a single workload. The LB comparisons have few runs, and the ARC-AGI-3 improvement isn’t the effect of the engine alone. Six months from now both the features and the numbers will have changed, so rather than trusting this table, it’s better to run the same checks on your own workload.

References#

ARC Prize https://arcprize.org/
Anthropic, Prompt caching https://platform.claude.com/docs/en/build-with-claude/prompt-caching
Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention (SOSP 2023) https://arxiv.org/abs/2309.06180
vLLM GitHub README https://github.com/vllm-project/vllm
PyTorch Foundation Welcomes vLLM as a Hosted Project (2025-05-06) https://pytorch.org/blog/pytorch-foundation-welcomes-vllm/
SGLang GitHub README https://github.com/sgl-project/sglang
Zheng et al., SGLang: Efficient Execution of Structured Language Model Programs https://arxiv.org/abs/2312.07104
vLLM v0.32.0 release notes https://github.com/vllm-project/vllm/releases/tag/v0.32.0
LMSYS Blog, SGLang v0.4 (2024-12-04) https://lmsys.org/blog/2024-12-04-sglang-v0-4/
SGLang v0.5.23 release notes https://github.com/sgl-project/sglang/releases/tag/v0.5.23
TechCrunch, Sources: Project SGLang spins out as RadixArk (2026-01-21) https://techcrunch.com/2026/01/21/sources-project-sglang-spins-out-as-radixark-with-400m-valuation-as-inference-market-explodes
vLLM documentation, Prefix Caching https://docs.vllm.ai/en/latest/design/prefix_caching/
Ding et al., CCL-Bench 1.0: A Trace-Based Benchmark for LLM Infrastructure (arXiv:2605.06544, Appendix G.1) https://arxiv.org/abs/2605.06544
LMSYS Blog, Deploying DeepSeek with PD Disaggregation and Large-Scale Expert Parallelism on 96 H100 GPUs (2025-05-05) https://lmsys.org/blog/2025-05-05-large-scale-ep/
Jarvislabs, SGLang vs vLLM: H100 Benchmarks, with TensorRT-LLM (2026-05-18) https://jarvislabs.ai/blog/vllm-sglang-trtllm-comparison
engchina, SGLang × Qwen3.8-27B × RTX 4090 ×4 (Qiita, in Japanese) https://qiita.com/engchina/items/d37f3169be4da2d31548
Daniel Franzen, ARC-AGI-3 solution https://github.com/da-fr/arc-agi-3-solution
Pennyroyal (SGLang for RTX PRO 6000) https://github.com/jpezzulli/sglang-rtxpro6000
TensorRT-LLM GitHub README https://github.com/NVIDIA/TensorRT-LLM
TensorRT-LLM Release Notes https://nvidia.github.io/TensorRT-LLM/release-notes.html
Text Generation Inference GitHub README https://github.com/huggingface/text-generation-inference
KTransformers GitHub https://github.com/kvcache-ai/ktransformers
NVIDIA Dynamo GitHub README https://github.com/ai-dynamo/dynamo
llm-d GitHub README https://github.com/llm-d/llm-d
AIBrix GitHub README https://github.com/vllm-project/aibrix
Ray Serve LLM documentation https://docs.ray.io/en/latest/serve/llm/index.html
LMDeploy GitHub README https://github.com/InternLM/lmdeploy
llama.cpp GitHub https://github.com/ggml-org/llama.cpp
Ollama GitHub https://github.com/ollama/ollama
MLC-LLM GitHub https://github.com/mlc-ai/mlc-llm