Unlocking Continuous Diffusion Language Models via Discrete Stochastic Localization

unlocking-continuous-diffusion-language-models-via-discrete-stochastic-localization

Unlocking Continuous Diffusion Language Models via Discrete Stochastic Localization

[Submitted on 18 Feb 2026 (v1), last revised 18 Nov 2026 (this version, v3)]

View PDF HTML (experimental)

Abstract:Continuous diffusion is a natural framework for non-autoregressive generation but has generally lagged behind masked discrete diffusion models (MDMs) on discrete sequence generation. We argue that continuous diffusion can effectively model discrete sequences through the joint design of noise parameterization and token representation geometry. We introduce emph{Discrete Stochastic Localization} (DSL), a continuous-state framework that combines a stochastic localization channel with unit-sphere token embeddings, resulting in a Bayes-optimal denoiser that depends only on the noisy state, without additional timestep conditioning. A single trained denoising network then supports multiple sampling methods, from continuous denoising to masked diffusion, random-order autoregressive generation and hybrid continuous-then-discrete generation. Parallel generation in this continuous-state framework, therefore, need not imply synchronized denoising: some tokens can establish context earlier while others remain open to refinement. Fine-tuning a pretrained MDLM checkpoint with DSL substantially improves generation quality, as measured by MAUVE, on OpenWebText across all step budgets from $T{=}128$ to $T{=}1024$. The same checkpoint supports random-order autoregressive sampling, and a hybrid continuous-then-discrete sampler using as few as $T=48$ total steps, without distillation or further training.

Submission history

From: Yunshu Wu [view email]
[v1] Wed, 18 Feb 2026 04:05:40 UTC (1,013 KB)
[v2] Wed, 20 May 2026 20:41:06 UTC (1,399 KB)
[v3] Wed, 18 Nov 2026 06:49:20 UTC (1,418 KB)