[Submitted on 14 Sep 2026 (v1), last revised 19 Oct 2026 (this version, v2)]
Abstract:Recurrent Transformers reusing their weights rather than stacking $L$ distinct layers are becoming widely adopted due to their parameter efficiency [1,2,3]. However, the exact representational and dynamical differences between looped and stacked architectures remain uncharacterized. This paper presents a controlled study on the example of bViT model [1] applying one weight-tied block $L$ times. We train two models: bViT and standard ViT [4] on 12-lead electrocardiogram (ECG) classification tasks from the PTB-XL dataset under identical training protocols. Despite an $8.9times$ parameter reduction, bViT achieves accuracy parity with ViT. Geometric similarity metrics demonstrate that both architectures construct comparable latent representations in an equivalent canonical order. Crucially, their dynamics differ: bViT exhibits smaller step sizes and inter-patient sensitivity, as well as near-neutral behavior away from the data manifold, whereas ViT exhibits collapsing dimensionality of representations and out-of-distribution feature expansion.
Submission history
From: Pawel Olszowiec [view email]
[v1] Mon, 14 Sep 2026 12:50:10 UTC (1,441 KB)
[v2] Mon, 19 Oct 2026 12:42:29 UTC (1,381 KB)
