[Submitted on 23 Oct 2025 (v1), last revised 8 Nov 2026 (this version, v3)]
Abstract:Stochastic gradient descent (SGD) is the workhorse of large-scale learning, yet classical analyses rely on assumptions that can be either too strong (bounded variance) or too coarse (uniform noise). The expected smoothness (ES) condition has emerged as a flexible alternative that ties the second moment of stochastic gradients to the objective value and the full gradient. This paper presents a self-contained convergence analysis of SGD under ES. We (i) refine ES with interpretations and sampling-dependent constants; (ii) derive bounds of the expectation of squared full gradient norm; and (iii) prove sublinear convergence rates with explicit residual errors for various step-size schedules. Furthermore, we empirically validate our theoretical framework by training a ResNet-18 model on the CIFAR-100 dataset. We uncover a two-phase dynamic in the gradient noise structure, transitioning from a suboptimality-dominated regime to a geometry-dominated regime. Our treatment unifies and extends recent theoretical threads in the literature.
Submission history
From: Yuta Kawamoto [view email]
[v1] Thu, 23 Oct 2025 14:39:57 UTC (2,897 KB)
[v2] Sun, 26 Oct 2025 02:53:43 UTC (2,897 KB)
[v3] Sun, 8 Nov 2026 12:23:55 UTC (156 KB)
