RMSNorm: Root Mean Square Layer Norm

Easy · Solved in PyTorch · Machine learning coding practice

Problem

Root Mean Square Layer Normalization (RMSNorm Zhang et al, 2019) is a simplified variant of LayerNorm that provides similar regularization benefits with lower computational overhead.

Unlike LayerNorm, RMSNorm does not re-center the activations (it doesn't subtract the mean). Instead, it only re-scales them by the root mean square of the features.

Topics: normalization, deep-neural-networks, statistics

The full statement, worked examples, hints and the test suite are available once you sign in. You can then solve RMSNorm: Root Mean Square Layer Norm in the browser and run it against the tests.

Related problems

• Batch Normalization Easy
• Layer Norm Easy
• Bradley Terry Ranking Medium
• Stateful Batch Normalization Medium
• Transformer Positional Encoding Easy
• Build Binary Classifier with Cross Entropy Loss Easy