Sliding Window Attention
Medium · Solved in PyTorch · Machine learning coding practice
Problem
Standard causal self-attention in autoregressive transformers has O(n^2) complexity where n is the sequence length, as each token attends to all previous tokens. This becomes prohibitively expensive for long sequences. Causal sliding window attention reduces this to O(n \times w) where w is a fixed window size, making it suitable for long-context language modeling and document processing.
Topics: decoding-only, inference, attention-mask
The full statement, worked examples, hints and the test suite are available once you sign in. You can then solve Sliding Window Attention in the browser and run it against the tests.