[Submitted on 13 Oct 2026 (v1), last revised 5 Nov 2026 (this version, v3)]
Abstract:Pixel-space diffusion models provide an appealing alternative to latent-space diffusion models by directly modeling raw images without relying on a learned tokenizer. However, scaling pixel-space models has predominantly followed dense strategies, tightly coupling model capacity with active computation. In this work, we show that standard LLM-style Mixture-of-Experts (MoE) architectures can be effectively transferred to pixel-space diffusion. Building on this finding, we introduce PixelMoE, a pixel-space generative framework that sparsely scales model capacity with standard LLM-style MoE, without diffusion-specific routing mechanisms or auxiliary routing objectives. Specifically, PixelMoE combines standard Token-Choice (TC) routing with fine-grained routed experts for input-adaptive sparse computation, a shared expert for capturing common representations, and a bias-based load-balancing strategy. To complement sparse capacity scaling, PixelMoE further introduces Pixel-Space Hierarchical Flow Matching, which varies spatial resolution along the denoising trajectory from global structure formation to fine-grained detail refinement. At comparable active parameter counts, PixelMoE consistently outperforms its dense counterparts across model scales, demonstrating effective sparse capacity scaling in pixel space. On class-conditional ImageNet-1K, PixelMoE achieves FID scores of 1.47 at 256×256 and 1.53 at 512×512 without representation alignment or perceptual losses, comparing favorably with leading pixel-space and latent-space generative models. Code and pretrained models are available at this https URL and this https URL, respectively.
Submission history
From: Ruixiao Dong [view email]
[v1] Tue, 13 Oct 2026 17:59:21 UTC (42,287 KB)
[v2] Thu, 29 Oct 2026 18:09:37 UTC (42,392 KB)
[v3] Thu, 5 Nov 2026 05:26:23 UTC (42,288 KB)
