Continuous Heavy-Tailed Theory of SGD

Event

Weekly DL Theory & Stat Phy Seminar

Short summary

In this seminar, I will talk about a recent line of works that propose to analyze SGD under heavy-tail noise assumptions using techniques from Levy-driven SDE theory and metastability analysis from statistical physics.

Papers

Papers discussed in the seminar:

• Mert Gürbüzbalaban, Umut Şimşekli, and Lingjiong Zhu. The Heavy-Tail Phenomenon in SGD. In arXiv 2020.
• Umut Şimşekli, Levent Sagun, and Mert Gürbüzbalaban. A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networks. In ICML 2019.