[Submitted on 29 Aug 2025 (v1), last revised 3 Nov 2026 (this version, v5)]
Abstract:Recent mechanistic work has uncovered learned algorithms within neural networks, from modular arithmetic to search and planning in game-playing agents. But does algorithmic structure guarantee algorithmic behavior? We investigate this in Leela Chess Zero, the strongest neural chess engine, where prior work identified learned look-ahead. By extending the logit lens to its move-selecting policy network, we discover that correct puzzle solutions–including immediate checkmates–often appear in intermediate layers but are systematically overridden in the final output, a phenomenon we term “forgotten puzzles”. Replicating prior analyses on these positions, we find that look-ahead operates normally–future moves of the correct continuation are represented, causally important, and linearly decodable–ruling out a failure of the algorithm. Instead, late layers increasingly shift toward safe play over aggression. To test whether this shift drives the override, we steer the model against these preferences and recover 71.7% of forgotten puzzles, providing causal evidence that safety priors override algorithmically computed solutions. These findings show that algorithmic structure does not guarantee algorithmic behavior: a model can internally solve a problem and still output the wrong answer.
Submission history
From: Elias Sandmann [view email]
[v1] Fri, 29 Aug 2025 07:51:45 UTC (4,230 KB)
[v2] Tue, 25 Nov 2025 13:55:48 UTC (13,555 KB)
[v3] Wed, 10 Jun 2026 10:31:06 UTC (38,855 KB)
[v4] Thu, 29 Oct 2026 12:22:54 UTC (37,512 KB)
[v5] Tue, 3 Nov 2026 12:56:46 UTC (37,512 KB)
