KL Divergence Penalty for RLHF

Easy · Solved in PyTorch · Machine learning coding practice

Problem

In Reinforcement Learning from Human Feedback (RLHF), a policy model \pi_\theta is fine-tuned to maximize a reward signal. Without constraints, the model can "drift" far from sensible language — producing outputs that game the reward but look nothing like real text. To prevent this, we add a KL divergence penalty that measures how far \pi_\theta has drifted from a frozen reference model \pi_{ref}.

Topics: reinforcement-learning, kl-divergence

The full statement, worked examples, hints and the test suite are available once you sign in. You can then solve KL Divergence Penalty for RLHF in the browser and run it against the tests.

Related problems

• DAPO: Clip-Higher Policy Optimization Loss Medium
• Group Relative Policy Optimization Loss Medium
• Transformer Positional Encoding Easy
• Batch Normalization Easy
• Build Binary Classifier with Cross Entropy Loss Easy
• Binary Quantization and Bit Packing Easy