[Submitted on 22 Oct 2026]
Authors:Zehao Lin, Qianfeng Shen, Xixuan Hao, Shaobo Cui, Chunyu Li, Kai Chen, Bo Tang, Zhenjun Wang, Jin Yan, Zhiyu Li, Feiyu Xiong
Abstract:Large language models increasingly act as controllers in structured decision pipelines, yet their decisions are still executed by autoregressive text decoding. Hierarchical RAG is one such setting: at each frontier of a document tree, an LLM commits to $b$ of $k$ sections under an input-dependent action grammar, executed as either failure-prone free-form JSON or reliable but strictly serial grammar-constrained generation. We show that this serialization is unnecessary for bounded-horizon control: the reachable future forms a shallow action-prefix tree, which we compile — one root prefill plus one tree-causal probe evaluate every prefix-conditioned action distribution with serial decoding’s tokens, positions, and causal ancestry. A prefix-state lemma makes compiled control policy-exact in exact arithmetic: the same policy with a two-forward critical path and zero output tokens. One further probe pass certifies the exact order-marginalized MAP set. Across four document-disjoint splits and two scales on the primary MetaX device, compiled control matches the serial policy on $84$–$89%$ of questions — every instrumented divergence a floating-point near-tie — and cuts control latency by $35$–$47%$ versus token-wise serial decoding and $6$–$7%$ versus an idealized chunked engine; an H800 repetition preserves the ordering.
Submission history
From: Zehao Lin [view email]
[v1] Thu, 22 Oct 2026 03:25:21 UTC (183 KB)
