From LoRA to QLoRA

Medium · Solved in PyTorch · Machine learning coding practice

Problem

In this first part, you will implement the forward pass of LoRA (Low-Rank Adaptation) (Hu et al., 2021), a groundbreaking parameter-efficient fine-tuning technique that reduces trainable parameters by up to 10,000x while maintaining model quality.

Topics: lora, low-rank-adaptation

The full statement, worked examples, hints and the test suite are available once you sign in. You can then solve From LoRA to QLoRA in the browser and run it against the tests.

More problems

• Optimizers: From Momentum to Adam Medium
• AdamW Optimizer Medium
• Implement Bagging Classifier with Decision Stumps Medium
• Basic KV Cache: Prefill and Decode Medium
• Binary Vector Scoring with Lookup Tables Medium
• Bradley Terry Ranking Medium