一、Adam与SGDM
Adam：fast training, large generalization gap,
unstable
• SGDM：stable, little generalization gap, better
convergence(?) 
二、Simply combine Adam with SGDM
SWATS [Keskar, et al., arXiv’17]
Begin with Adam(fast), end with SGDM
Start
training
Adam
Meet some
criteria
Convergence
三、Towards Improving SGDM
Adaptive learning rate algorithms：dynamically
adjust learning rate over time
• SGD-type algorithms：fix learning rate for all
updates… too slow for small learning rates and
bad result for large learning rates
四、Advices
SGDM：Computer vision
Adam
• NLP
• Speech synthesis
• GAN
• Reinforcement learning
