Coordinate Prefill-to-Decode KV Handoff

Medium · Solved in PyTorch · Machine learning coding practice

Problem

Implement a small event-driven handoff model; it does not transfer real tensors or reproduce a specific connector. Track num_requests independently, initially state NEW=0 with owner NONE=0. States are PREFILL=1, READY=2, LOADING=3, DECODING=4, FINISHED=5, CANCELLED=6. Owners are NONE=0, PREFILL=1, DECODE=2.

Topics: inference, vllm, pytorch

The full statement, worked examples, hints and the test suite are available once you sign in. You can then solve Coordinate Prefill-to-Decode KV Handoff in the browser and run it against the tests.

Related problems

• Mask Logits with a Grammar State Easy
• Pack Prefill and Decode Batches Medium
• Continuous Batching and Preemption Hard
• Measure Token Latency and SLO Goodput Medium
• Route Requests by Replica Load Easy
• Schedule Decode and Prefill Within a Token Budget Medium