Measure Token Latency and SLO Goodput
Medium · Solved in PyTorch · Machine learning coding practice
Problem
Given completed request traces, return a float64 vector [mean_ttft, mean_itl, token_throughput, request_goodput]. The float64 arrivals: (R,) gives arrival times; Python list token_times contains one nonempty, nondecreasing list of token emission times per request. Times are in seconds and each emission is at or after its request's arrival.
TTFT is first emission minus arrival; average it across requests. ITL is each successive emission gap; average all gaps pooled across requests, or return 0 if none exist. Throughput is total emitted tokens divided by window_end - window_start.
Topics: inference, vllm, pytorch
The full statement, worked examples, hints and the test suite are available once you sign in. You can then solve Measure Token Latency and SLO Goodput in the browser and run it against the tests.