[Submitted on 27 Sep 2026 (v1), last revised 15 Oct 2026 (this version, v3)]
Authors:Yunao Zheng, Bin Wen, Xiaojie Wang, Kaiyu Jiang, Xuanyu Zheng, Changyi Liu, Hongyi Fu, Jianxiong Wang, Tianke Zhang, Haonan Fan, Yingxin Li, Jiankang Chen, Xu Wang, Tingting Gao, Han Li
Abstract:Improving language model efficiency requires not only cheaper operations but also eliminating redundant computation. We argue that effective modeling requires information accessibility and computational depth, but not repeated historical retrieval across layers. Guided by this principle, we introduce the Global State Model (GSM), a causal encoder–decoder architecture that separates historical information acquisition from deep computation. The encoder progressively aggregates long-range information through multi-stage sparse retrieval into a fixed-size shared state, while decoder layers perform successive transformations over this state without repeated historical access. This design makes decoder attention and cache costs independent of history length, while dependency-aware pruning further reduces prefill computation. Experiments across multiple model scales show that GSM achieves downstream performance comparable to matched DeepSeek-V4.1-Flash-based baselines, with improvements at some scales, while preserving long-context retrieval capabilities. GSM also improves training and inference throughput and reduces cache overhead. These results demonstrate that eliminating architectural redundancy offers an effective path toward efficient language modeling without sacrificing information accessibility or computational depth.
Submission history
From: Yunao Zheng [view email]
[v1] Sun, 27 Sep 2026 11:19:21 UTC (1,503 KB)
[v2] Tue, 13 Oct 2026 03:26:20 UTC (1,538 KB)
[v3] Thu, 15 Oct 2026 03:28:02 UTC (1,479 KB)
