[Submitted on 13 Oct 2026]
Abstract:Transformer language models develop attention heads that specialize in syntactic relations, for example, attending from verbs to their subjects. Do these heads play a causal role in the model’s syntactic predictions? Evidence for such a role is mixed to negative: even when we force these heads to attend to a distractor noun other than the subject, the model’s syntactic predictions are still more consistent with the subject than with the distractor. Interventions on specialized syntax heads had a significantly stronger effect on syntactic predictions than interventions on random heads, but only in one of the two measures we propose.
Submission history
From: William Timkey [view email]
[v1] Tue, 13 Oct 2026 21:47:26 UTC (116 KB)
