[Submitted on 5 Oct 2026]
Authors:Zixuan Huang (1 and 2), Tianhao You (1), Guozhang Chen (1, 3 and 4) ((1) School of Computer Science, Peking University, Beijing, China, (2) School of Physics, Peking University, Beijing, China, (3) State Key Laboratory of Multimedia Information Processing, Peking University, Beijing, China, (4) National Engineering Research Center of Visual Technology, Peking University, Beijing, China)
Abstract:Diffusion transformers have achieved remarkable success in image generation, but accelerating attention while preserving generation quality remains a major challenge at high resolutions. Inspired by statistical physics and the widespread occurrence of scale-free laws in nature, we introduce Scale-Free Attention (SFA), which preserves fine local interactions while summarizing distant regions at progressively coarser scales. For $N$ tokens, mean pooling, token-count correction, and a shared softmax over a scale-free spatial hierarchy give SFA $O(Nlog N)$ complexity in both forward and backward passes. Our Triton implementation achieves $357.3times$ forward and $265.6times$ backward attention speedups over FlashAttention at 262,144 tokens. After 40 epochs of PixelFlow training from scratch on ImageNet at $256times256$ resolution, SFA achieves the best generation quality among five attention mechanisms, including full attention. Fréchet inception distance evaluated on 50,000 generated samples (FID-50k) improves from 13.25 for full attention to 12.51, and the advantage over full attention persists through 80 epochs. SFA also achieves the best generation quality among the compared efficient-attention methods on Flickr-Faces-HQ (FFHQ). Beyond training from scratch, Stable Diffusion 3.5 Medium with attention replaced by SFA retains the generation quality of its full-attention counterpart under matched fine-tuning budgets. We obtain similar results when fine-tuning pretrained PixelFlow on ImageNet. SFA establishes scale-free structure as an effective design principle for efficient, high-quality diffusion transformers.
Submission history
From: Zixuan Huang [view email]
[v1] Mon, 5 Oct 2026 07:22:18 UTC (29,507 KB)
