vLLM's PagedAttention: How Memory Management Supports Large Model Inference

Technical Teardown: Analyzing AI Technology Frameworks — Explanation, Analysis, Technical Assessment, Value Judgment, and Implementation. Author: Yongliang

---

In 2023, there was a universally acknowledged pain point in large model inference: GPU VRAM looked large enough, but when you actually ran it, you couldn’t crank up the batch size. The reason wasn’t the model weights—weights are static, you calculate once and know how much space they take—but the KV cache: it constantly expands as the conversation lengthens, and every request grows differently. In that year, UC Berkeley’s Sky Computing Lab provided an answer that would be cited repeatedly: move the operating system’s virtual memory paging into attention calculation, and you get PagedAttention, along with the serving engine behind it, vLLM.

More than two years later, vLLM has grown from a SOSP paper into the de facto standard for inference serving: as of September 2026, it has approximately 92,000 stars on GitHub, about 415,000 weekly downloads on PyPI, and over 2,000 contributors. This article, as always, breaks down six things: what it is, the core mechanism down to the source code level, how to evaluate the data, how to choose between it and SGLang, whether it’s worth using, how to implement it, and—if you want to write a similar memory management system yourself—what the minimal thought process is.

1. What is this?

One-sentence positioning: vLLM is a high-throughput, memory-efficient LLM inference and serving engine (original GitHub repository description). The core problem it solves is that the KV cache takes up the bulk of inference VRAM; if management is crude, it leads to waste, waste suppresses batch size, and batch size directly determines throughput.

Key facts: The repository was created in February 2023, originally from UC Berkeley’s Sky Computing Lab; the paper “Efficient Memory Management for Large Language Model Serving with PagedAttention” was published at SOSP 2023, a top conference in the systems field, arXiv ID 2309.06180. Among the nine authors are heavyweights in the distributed systems field like Ion Stoica and Joseph Gonzalez; license is Apache-2.0; approximately 91,598 stars, 22,113 forks. The README states it is “maintained by a community of dozens of academic institutions and companies,” and the contributor list is ranked by commit volume, with the top 100 developers contributing a total of over 12,000 commits—the high star count isn’t fake; it’s a real team investing long-term.

2. Core Mechanisms

2.1 First, let’s do the math: How much VRAM does KV cache actually consume?

PagedAttention wasn’t a gimmick conjured out of thin air; it came after doing the math clearly. This math is still worth memorizing for everyone doing inference today:

KV cache bytes per token = 2 × layers × KV heads × head dimension × bytes per parameter (multiplied by 2 because there is one copy each for K and V).

Plugging in the official configuration of Qwen2.5-72B (Hugging Face config.json): 80 layers, GQA architecture with 8 KV heads, head dimension 128, fp16 precision at two bytes—each token takes up 327,680 bytes, about 0.31 MB. A single request with an 8k context needs 2.5 GB just for KV cache; one hundred concurrent requests means 250 GB. Here, GQA plays a huge role: 64 attention heads share 8 KV heads; if it were the old multi-head attention architecture, this number would be eight times larger.

This math explains why the three types of waste in the paper are fatal. Reservation: Old systems allocated a full stretch of contiguous VRAM for each request based on the estimated maximum length, leaving most positions empty. Internal fragmentation: The actual length doesn’t reach the estimated length, so the tail is wasted. External fragmentation: VRAM is cut into holes of varying sizes, unable to piece together a large contiguous chunk—old systems required contiguous VRAM, a pit they dug for themselves. The experimental conclusion in the paper is: in old systems, the effective utilization rate of KV cache could be as low as about 20%, with 80% of VRAM occupied for nothing. Since serving is a memory-constrained scenario, the direct consequence of occupying for nothing is that the batch size can’t be increased, and throughput doesn’t go up.

2.2 PagedAttention: Three components of OS paging ideas

The idea behind PagedAttention is shockingly straightforward: operating systems manage virtual memory by slicing the continuous address space seen by processes into fixed-size pages, relying on a page table to map to physical memory—why not manage KV cache the same way?

In implementation, it’s a three-piece set. First, paging: KV cache is no longer reserved as a whole continuous space per request, but sliced into fixed-size blocks (commonly 16 tokens per block in paper experiments), and a block is allocated only when needed. Second, block table: Each request maintains a mapping table from logical block numbers to physical block numbers. The attention kernel gathers corresponding physical blocks according to the table; physical blocks are not required to be contiguous. This step is the key to the entire article—the “continuous” assumption is exactly the source of external fragmentation, and the block table eliminates it completely. Third, on-demand allocation: A new block is applied for only after the current block is filled from left to right. The waste per request is compressed to within one block, approaching zero waste (paper metric).

2.3 Copy-on-write: Reference counting for shared blocks

Paging also brought an unplanned benefit: sharing. Parallel sampling (generating n candidates from the same prompt), multiple paths in beam search—the prefix KV is completely identical—old systems either stored multiple copies or relied on special logic for hard sharing. vLLM’s approach is exactly the same as the operating system: increment the reference count for shared blocks; if someone wants to write and the reference count is greater than one, copy first then write (copy-on-write). The result is that prefix KV can be reused both within and between requests. The paper lists this as PagedAttention’s second major contribution: besides near-zero waste, add flexible sharing.

2.4 Source code level: V1’s block pool and hash chain

In 2025, vLLM completed the V1 engine rewrite, and now all scheduling logic is under the vllm/v1 directory; the old engine code has been deleted. The entry point for block management is BlockPool in vllm/v1/core/block_pool.py, which manages two structures:

The first is the free block queue free_block_queue—a doubly linked list sorted by “most recently used”. Released blocks that have been reused go to the back of the queue; new allocations take from the front; if the block at the front still carries prefix cache content, it is evicted from the cache before allocation. LRU eviction doesn’t require any additional data structure; the queue order itself is the eviction order. The second is the hash-to-block mapping cached_block_hash_to_block, supporting prefix cache lookups.

The block’s hash isn’t a simple “hash of this block’s content,” but chained. hash_block_tokens in kv_cache_utils.py looks like this: sha256(parent block hash, this block’s token sequence, extra key). The parent block hash acts as a salt—as long as the prefix content differs by one token, the hash of the entire subsequent chain is different, and different prefixes cannot collide with the same key; the function itself carries an LRU cache, so the same block content doesn’t need to be recalculated. When a new request comes in, get_computed_blocks in kv_cache_manager.py compares its own prompt against this hash chain block by block; hits increment the reference count and take over directly, skipping the prefill for the corresponding part entirely.

There’s a detail in the source code worth mentioning: even if every block of the prompt hits the cache, the last token must be recalculated—because getting logits relies on live calculation (source code comment: When all tokens hit the cache, we must recompute the last token to obtain logits). This small design of “calculate one step even on a full hit” ensures that cache hits don’t change output semantics.

2.5 Scheduler: One loop to handle prefill, decode, and preemption

The scheduler entry is at vllm/v1/core/sched/scheduler.py. At the top of this file, there is a comment by the author Woosuk that is worth reading in full: there is no distinction between a “decode phase” and a “prefill phase” in the scheduler; each request only maintains two numbers—num_computed_tokens (number of tokens computed) and num_tokens_with_spec (position to compute to, equal to prompt length plus generated length plus speculative tokens). What the scheduler does every step is allocate token quotas for each request, helping the former catch up to the latter.

The benefit of this abstraction is that three scenarios are unified by the same loop: long prompts are sliced and mixed into the batch (chunked prefill, so one long request doesn’t starve a bunch of decodes); requests with prefix cache hits continue directly from the middle; speculative decoding calculating a few more candidate tokens is just num_tokens_with_spec adding a segment. The total quota per step is capped by max_num_scheduled_tokens.

The means when memory is insufficient is preemption: _preempt_request pulls requests from the tail of the running queue back to the waiting queue, releasing all their blocks and encoder cache. V1 preemption only has one method: recompute—when a pulled request gets its turn again, it prefills from the beginning, without swapping out to CPU memory. This is a clear trade-off: recompute wastes compute power, but implementation is simple, and it avoids the jitter of moving KV cache between CPU and GPU. In real online systems, the number of preemptions is a metric to watch—frequent preemption indicates capacity configuration or scheduling parameter issues.

V1 also implements two levels of overlap: output processing (detokenize, streaming send) overlaps with GPU computation; in asynchronous scheduling mode, preparing the next step’s scheduling decision overlaps with the current step’s computation. The scheduler is pure Python, and this kind of overlap is its free lunch.

2.6 From block table to GPU: Kernels, backends, and multi-processing

The block table ultimately needs to become something on the GPU: every scheduling step passes the block_table to the attention kernel as a tensor. Decode uses vLLM’s own paged attention kernel hand-written in CUDA/CUTLASS under csrc/attention, gathering physical blocks according to the table; prefill has a different attention shape (calculating the entire prompt at once), so it doesn’t take this path, connecting directly to existing backends like FlashAttention and FlashInfer. Above both is the AttentionBackend abstraction layer; NVIDIA, AMD, CPU, and TPU each implement their own backends, and kernel details are invisible to the scheduler.

The decode step shape is highly regular (exactly one more token per request per step), and V1 captures the entire decode step using CUDA Graph, eliminating launch overhead between Python and CUDA; scenarios where prefill and decode are mixed rely on piecewise compilation to leave the shape-changing parts outside the graph. This step complements paging: precisely because each request increases or decreases memory by blocks, who stays and who leaves in the batch doesn’t affect others’ memory layout, so the entire step’s calculation can be stably captured into a graph.

Finally, the multi-process skeleton: the API server process receives HTTP requests, does tokenization and multimodal loading, and communicates via ZMQ with the engine core (many-to-many topology of multiple API servers to multiple engine cores); the engine core process makes scheduling decisions; GPU worker processes are one per card, the quantity equals DP×PP×TP, and are only responsible for running the forward pass; if data parallelism is enabled, there is also a DP coordinator for load balancing. The standard deployment for a single-machine 4-card tensor parallel is 1 API server + 1 engine core + 4 workers, totaling 6 processes.

3. Technical Evaluation: 2-4x is the paper’s metric; first, look at who it’s being compared to

The paper’s data: at the same latency level, vLLM’s throughput is 2-4 times higher than the strongest systems at the time, FasterTransformer and Orca; the longer the sequence, the larger the model, and the more complex the decoding algorithm, the more obvious the advantage (original paper text). Note the baseline—this is a 2023 comparison. Orca was already the most advanced iteration-level scheduling system at the time. vLLM won by memory management, not operators.

My interpretation is on two levels. The first level, the paper’s measurement that “old system effective memory is as low as about 20%” is more fundamental than the 2-4x figure—it explains why the improvement holds: compressing waste from 80% to within one block allows the same card to fit several times more requests, so throughput naturally doubles. This causality is hard. The second level, in the coordinate system of 2026, the 2-4x figure can no longer be copied as a slogan; today’s opponents are SGLang, TensorRT-LLM, and the like, and the gap has shrunk to a workload-related tens of percent.

Popularity can be seen in three numbers: about 92,000 stars; the top 100 contributors have a total of over 12,000 commits; PyPI weekly downloads are about 415,000. These three numbers interlock, and the scale of real use is unquestionable—it is the number one infrastructure in the inference serving space.

A dose of cold water is needed too. Benchmarks are paper metrics; self-reproduction depends on your own request length distribution and concurrency patterns; prefix caching has limited benefits for short requests and high-diversity prompt scenarios, just taking up VRAM for nothing; V1’s multi-process architecture has CPU overhead for single-card small models, and the official documentation itself suggests configuring CPU resources according to the number of GPUs.

4. How to choose between vLLM and SGLang

SGLang profile in one sentence: A high-performance serving framework from LMSYS, created in January 2024, with about 36,000 stars, Apache-2.0. The core differentiation is RadixAttention—organizing prefix cache into a radix tree instead of a hash table. The official claim is “up to 5x inference acceleration” (January 2024 official blog, self-rated data). In the past year, it has been widely adopted in agent workflows and RL training frameworks (verl, slime, etc.).

Three sentences for selection: default to vLLM if there is no clear reason—it has the most complete ecosystem, documentation, and hardware adaptation (first-class support for NVIDIA/AMD/Intel/CPU, plugins for TPU/Gaudi/Ascend/Apple Silicon), and the lowest search cost for pitfalls; if your workload is multi-turn agent calls, complex structured output, or needs to be embedded in RL training loops, it’s worth pulling in SGLang for head-to-head stress testing, its aggressive optimizations have measured advantages in these scenarios; both are Apache-2.0, migration cost isn’t high, let your own traffic do the talking.

Philosophically, they share the same origin: RadixAttention and PagedAttention’s prefix sharing are two data structures for the same line of thought (radix tree vs. block table hash), and the SGLang project itself reuses a large amount of vLLM’s infrastructure. This isn’t a zero-sum game; it’s the entire industry converging on the consensus that “KV cache is a reusable resource.”

5. Value Judgment

The real problem: inference cost equals memory efficiency times batch size. vLLM changed KV cache from a “reservation system” to a “paging system.” In my view, this is the most important single improvement in the inference stack since 2023—it doesn’t change the model, doesn’t change the chip, and purely relies on system software to squeeze several times more throughput out of the same hardware. Today almost all open-source inference frameworks use its block management idea, which in itself is proof of value.

The boundaries are also clear. Running small models on a single machine with single-digit concurrency, the benefits from paging and continuous batching are limited, and a simple solution might be more worry-free; it doesn’t solve time-to-first-token latency—the prefill compute bottleneck relies on kernels and parallel strategies, that’s another battlefield; support for non-standard architectures (state space models like Mamba) has just grown the HybridKVCacheCoordinator layer in V1 and is still evolving. When to use it: any scenario where you seriously need to provide LLM services to the outside, it is the default starting point. When not to use it: running a few pieces of data offline once, or using it in training—that’s other tools’ territory.

6. How to Deploy

Installation in one line:

uv pip install vllm    # or pip install vllm

Starting the service is also one line. vllm serve starts an OpenAI-compatible interface (also supporting the Anthropic Messages API and gRPC):

vllm serve Qwen/Qwen3-32B --tensor-parallel-size 2

Offline batch inference uses the Python LLM class, results in a few lines; supports over 200 model architectures—decoder-only, MoE, hybrid attention/state space, multimodal, embedding, reward models can all be served, covering parallel sampling, beam search, structured output (xgrammar/guidance), tool calling parsing, multi-LoRA. On the distributed side, there are five parallelisms configurable: tensor, pipeline, data, expert, and context. Two implementation suggestions: decide whether to turn on prefix caching based on traffic characteristics—turn it on for multi-turn dialogue loads, turn it off for short request high-diversity loads; in production, put KV cache utilization and preemption count into monitoring; these two indicators expose capacity problems earlier than GPU utilization.

7. How to build a similar solution yourself

“Building your own set” isn’t a hypothetical question. nano-vllm on GitHub (GeeeekExplorer/nano-vllm, open sourced June 2025, MIT, about 15,000 stars) is a minimal readable replica written by the community after reading vLLM source code. The entire engine is on the order of a thousand lines—it proves one thing: the core idea of PagedAttention can be explained on one page. Six steps for the skeleton:

1. Block pool plus block table: At startup, slice the KV cache VRAM into a fixed-size physical block pool; each request maintains a table of logical blocks to physical blocks, and attention gathers according to the table.

# Minimal skeleton: block pool
free_blocks = deque(range(num_gpu_blocks))   # Physical block free queue
block_table = {}                             # req_id -> [physical block number]

2. On-demand allocation: After each decode step, check if the current block is full; only when full, take a block from the free queue and hang it—waste naturally does not exceed one block.

3. Reference count plus copy-on-write: Increment refcount for shared blocks; if refcount is greater than one before writing, copy first then write. Sharing for parallel sampling and beam search relies on this one rule.

4. Scheduler loop only chases two numbers: Each request records num_computed and num_total; each step picks requests within the token budget, letting computed catch up to total; whoever finishes exits, if it doesn’t fit, preempt by priority (pull back request, release blocks, recompute prefill from scratch later). Chunked prefill, prefix cache, speculative decoding are all just special cases of this loop.

5. Prefix hash cache: Build a chained hash based on sha256(parent block hash + this block token), compare block by block to hit and skip prefill; the head of the free queue is the eviction candidate.

6. Don’t write kernels yourself: Connect attention directly to FlashAttention; leave CUDA/CUTLASS for when you really have a performance team. nano-vllm does exactly this.

These six steps add up to a few hundred lines of Python plus existing kernels to get it working—the core insight of the PagedAttention paper was never complicated; what’s hard is maintaining it as production-grade day after day for two years. This is also why vLLM source code is more worth reading than the paper: architectural ideas can be explained on one page, but engineering completeness is the moat.

Conclusion

PagedAttention’s contribution can be condensed into one sentence: KV cache is not “one continuous array per request,” but a “pooled resource allocated on demand and shared by blocks.” vLLM used the operating system’s old method to solve the new problem of large model inference, becoming the de facto standard on this one thing. For most teams, selection doesn’t require hesitation—default to vLLM, pull in SGLang for head-to-head stress testing only if you have special workloads; for engineers who want to build inference systems, its source code is a better textbook than the paper.

References

• vLLM GitHub repository (README, architecture documentation): https://github.com/vllm-project/vllm
• Paper: Efficient Memory Management for Large Language Model Serving with PagedAttention, SOSP 2023, arXiv:2309.06180 (Abstract, §2 Waste Analysis, §6 Evaluation)
• vLLM official architecture documentation Architecture Overview (docs.vllm.ai, V1 multi-process architecture and source code index)
• vLLM V1 source code: vllm/v1/core/block_pool.py (block pool and free queue), vllm/v1/core/kv_cache_utils.py (hash_block_tokens chained hash), vllm/v1/core/kv_cache_manager.py (get_computed_blocks prefix hit), vllm/v1/core/sched/scheduler.py (scheduler main loop and _preempt_request preemption), csrc/attention (paged attention CUDA/CUTLASS kernels)
• Qwen2.5-72B model configuration (Hugging Face config.json: 80 layers / GQA 8 KV heads / head dimension 128, basis for KV cache VRAM calculation)
• GitHub repository metadata and contributor list, pypistats.org vllM downloads (September 2026)
• nano-vllm minimal replica (GitHub: GeeeekExplorer/nano-vllm, MIT): https://github.com/GeeeekExplorer/nano-vllm
• Source code interpretation secondary materials (for cross-reference): CNBlogs “Nano-vLLM Source Code Interpretation” series, Juejin “nano-vllm’s KV Cache and Paged Attention” series, CSDN “Large Model Inference Engine vLLM Study Notes” series (V1 Architecture and Prefix Caching chapters)
• SGLang GitHub repository (README and metadata): https://github.com/sgl-project/sglang; LMSYS blog 2024-01-17 (RadixAttention official statement)