[Submitted on 7 Jul 2026 (v1), last revised 21 Oct 2026 (this version, v3)]
Abstract:Speculative decoding greatly increases the interactivity of autoregressive language models by trading off computation for extra tokens generated in a single forward pass. Factorized draft models are especially efficient because they predict future-token marginals in parallel, but their independence assumption causes acceptance rates to degrade sharply as the speculative budget grows. We analyze this limitation and introduce Weaver, a lightweight autoregressive adapter that constructs proposal trees from the top-K marginals of a factorized drafter. Weaver restores conditional dependencies between proposed tokens while avoiding a full-vocabulary projection. To support fast verification for models with Gated Delta Net layers, we derive a rollback-free tree-verification algorithm and implement optimized CUDA kernels in SGLang. By combining these model and systems contributions we achieve a 4.08-fold speedup over autoregressive decoding, and outperform a highly optimized DFlash baseline by 16.3%.
Submission history
From: Yuma Oda [view email]
[v1] Tue, 7 Jul 2026 19:48:36 UTC (956 KB)
[v2] Sun, 12 Jul 2026 19:52:09 UTC (911 KB)
[v3] Wed, 21 Oct 2026 16:07:58 UTC (908 KB)
