K² · Artificial intelligence

Nobody Has Proven the World's Favorite AI Optimizer Actually Works

The paper studies whether AdamW, the de facto optimizer for training large language models, can converge under heavy-tailed gradient noise.

Dingzhi Yu, Hongyi Tao, Yuanyu Wan et al.

5 authors · cs.LG, cs.AI, math.OC, stat.ML

arXiv preprintArtificial intelligenceJun 2026 · ~65s read

The 30-second scan

The algorithm that trains almost every large language model has never been proven to work under the conditions it's actually used in.

The authors note that existing AdamW theory lives mostly in finite-variance regimes, which is unsatisfying because empirical evidence indicates stochastic gradient noise in LLM pretraining is typically heavy-tailed.
The abstract states that recent work shows sign-based optimizers such as Lion and Muon achieve sharp heavy-tailed rates, and that AdaGrad can also converge under heavy-tailed noise.
The authors state that no rigorous convergence theory for AdamW has yet been established in the heavy-tailed regime.