[Submitted on 16 Sep 2026 (v1), last revised 27 Oct 2026 (this version, v2)]
Abstract:Warm-start transfer accelerates delayed generalization (grokking) on algorithmic tasks, yet which architectural components drive this acceleration and whether transferred circuits remain stable under continued optimization remain unclear. Across a scale-matched 108-run factorial battery ($N=12$ paired seeds), transferring internal attention/MLP matrices ($B$) alongside representations ($E+U$) lifts early accuracy by $5.46$ percentage points and cuts validation confirmation latency by $617$ steps (Holm $p=0.0039$). Whereas readout plus internal blocks ($U+B$) matches full transfer within $pm 500$ steps in 1-layer models (TOST $p=2.91times 10^{-13}$), a 2-layer replication confirms internal-block acceleration ($+704.67$ units, $p=4.88times 10^{-4}$; $8.1times$ faster trigger) while exposing an inter-block interface where omitting embeddings incurs a $-4{,}475.6$-unit deficit (TOST $p=1.00$). Under continued optimization, unshielded runs suffer severe post-grokking relapse. Freezing carriers ($E+U$) suppresses 1-layer drawdown ($22.06% to 0.60%$ on $2a+b$; $28.56% to 5.09%$ on $3a+b$; $11.09% to 0.15%$ on $a^2+b^2$), but leaves internal blocks vulnerable at high step sizes. Dissecting 12 post-trigger schedules shows that Curvature-Preconditioned Bounded Step-Budget Consolidation ($S_H approx 0.02, lambda=1.0$, preserved $v_t$)—via 201-step micro-stepping ($86.6%$ lower active compute) or exponential decay ($tau=200$)—compresses drawdown to $0.08%$–$0.17%$ on 1-layer $2a+b/3a+b$, $0.13%$–$0.20%$ on 2-layer $2a+b$, $0.02%$ ($66.81% to 0.02%$, Holm $p=0.0020$) on unseen pure-FP32 2-layer subtraction ($a-b$), and $0.09%$–$0.18%$ (Holm $p=0.031$) on non-abelian $S_5$ where carrier freezing fails ($10.73%$ drawdown, $p=0.273$).
Submission history
From: Zeyu Jia [view email]
[v1] Wed, 16 Sep 2026 03:28:39 UTC (1,210 KB)
[v2] Tue, 27 Oct 2026 02:46:00 UTC (485 KB)
