Build Causal Grouped-Query Attention (GQA)

Hard · Solved in PyTorch · Machine learning coding practice

Problem

Grouped-Query Attention (Ainslie et al., 2023) is the "Goldilocks" solution between Multi-Head Attention (MHA) and Multi-Query Attention (MQA), balancing memory efficiency with model quality.

Topics: transformers, attention, efficiency

The full statement, worked examples, hints and the test suite are available once you sign in. You can then solve Build Causal Grouped-Query Attention (GQA) in the browser and run it against the tests.

Related problems

• Build Causal Multi-Query Attention (MQA) Hard
• Calculate Self-Attention Easy
• MoE: Dispatch and Weighted Combine Medium
• MoE: Router Load-Balancing Loss Medium
• MoE: Top-k Expert Routing Easy
• Build Causal Multi-Head Attention for nanoGPT Hard