Basic KV Cache: Prefill and Decode
Medium · Solved in PyTorch · Machine learning coding practice
Problem
In LLM inference, we distinguish between two execution modes:
Prefill Phase: Processes the entire input prompt at once and populates the KV Cache. This is highly parallel.
Decoding Phase: Generates tokens one-by-one. Each step only computes K and V for the newest token and reuses the cache for previous tokens.
Topics: transformer, llm, inference
The full statement, worked examples, hints and the test suite are available once you sign in. You can then solve Basic KV Cache: Prefill and Decode in the browser and run it against the tests.