Stateful Batch Normalization

Medium · Solved in PyTorch · Machine learning coding practice

Problem

In a real world training loop, Batch Normalization is not a pure function. It must maintain a running state so that it can perform consistently during inference when a batch might consist of only a single sample.

Topics: normalization, neural-networks, state-management

The full statement, worked examples, hints and the test suite are available once you sign in. You can then solve Stateful Batch Normalization in the browser and run it against the tests.

Related problems

• Batch Normalization Easy
• Layer Norm Easy
• RMSNorm: Root Mean Square Layer Norm Easy
• Optimizers: From Momentum to Adam Medium
• AdamW Optimizer Medium
• Implement Bagging Classifier with Decision Stumps Medium