[Submitted on 12 Mar 2026 (v1), last revised 15 Oct 2026 (this version, v2)]
Abstract:Rotary Positional Embedding (RoPE) is widely used in transformers to encode relative positional information, yet the effect of applying RoPE to only a subset of hidden dimensions remains underexplored. This design choice can substantially reduce memory usage, especially at long context lengths. We show that partial RoPE achieves up to 10$times$ memory savings over standard RoPE cache while maintaining comparable final loss. We systematically study its impact on training dynamics and convergence across architectures, model sizes, sequence lengths, and datasets. Our results reveal that: (1) applying RoPE to only a small fraction of dimensions (around 10%) matches the convergence of full RoPE; (2) these trends remain consistent across settings; and (3) NoPE (No Positional Encoding) can produce loss spikes, which are mitigated by minimal RoPE usage or QK-Norm application. Overall, our findings highlight partial RoPE as an effective trade-off between efficiency and stability.
Submission history
From: Mohammad Aflah Khan [view email]
[v1] Thu, 12 Mar 2026 07:05:13 UTC (3,864 KB)
[v2] Thu, 15 Oct 2026 08:33:35 UTC (3,883 KB)
