PagedAttention: KV Cache Block Manager
Medium · Solved in Python · Machine learning coding practice
Problem
In systems like vLLM, the KV cache is managed similarly to an Operating System's virtual memory. A request consists of a continuous stream of tokens, divided into Logical Blocks.
The system uses a Block Table to map these logical blocks to scattered Physical Block Pool in GPU memory.
Topics: memory-management, inference, data-structures
The full statement, worked examples, hints and the test suite are available once you sign in. You can then solve PagedAttention: KV Cache Block Manager in the browser and run it against the tests.