Optimizers: From Momentum to Adam

Medium · Solved in PyTorch · Machine learning coding practice

Problem

In this first part of the optimizer series, you will implement SGD with Momentum (Qian, 199900116-6)), a fundamental enhancement to vanilla Stochastic Gradient Descent that dramatically improves convergence speed.

Topics: optimization, momentum

The full statement, worked examples, hints and the test suite are available once you sign in. You can then solve Optimizers: From Momentum to Adam in the browser and run it against the tests.

Related problems

• AdamW Optimizer Medium
• Bradley Terry Ranking Medium
• Linear Regression: From OLS to Lasso Medium
• Speculative Decoding: Rejection Sampling Hard
• Implement Bagging Classifier with Decision Stumps Medium
• Basic KV Cache: Prefill and Decode Medium