Why 0.9? Towards Better Momentum Strategies in Deep Learning

 By Medium - 2021-02-26

Description

Momentum is a widely-used strategy for accelerating the convergence of gradient-based optimization techniques. Momentum was designed to speed up learning in directions of low curvature, without…

Summary

  • Towards Better Momentum Strategies in Deep Learning.
  • In deep learning, most practitioners set the value of momentum to 0.9 without attempting to further tune this hyperparameter (i.e., this is the default value for momentum in many popular deep learning packages).
  • Demon in Code The code for implementing the Demon schedule is also extremely simple.
  • Demon performance is depicted in the top row, while the performance of vanilla optimizer counterparts (i.e., SGDM and Adam) are depicted in the bottom row.

 

Topics

  1. Machine_Learning (0.42)
  2. Backend (0.17)
  3. NLP (0.12)

Similar Articles

Machine Learning Optimization Methods and Techniques

By Medium - 2020-12-07

The principal goal of machine learning is to create a model that performs well and gives accurate predictions in a particular set of cases. In order to achieve that, we need machine learning…

CLIP: Connecting Text and Images

By OpenAI - 2021-01-05

We’re introducing a neural network called CLIP which efficiently learns visual concepts from natural language supervision.