[Submitted on 24 Mar 2026 (v1), last revised 3 Nov 2026 (this version, v3)]
Abstract:The increasing memory demand of the Key-Value (KV) cache poses a significant bottleneck for Large Language Models (LLMs) in long-context applications. Existing low-rank KV compression methods reduce this footprint by modifying model projections, limiting the flexibility to switch back to standard full-cache inference when sufficient memory is available. In this paper, we propose EchoKV, a flexible KV cache compression framework that supports on-demand transitions from full KV caching to compressed caching. Unlike traditional compression-decompression paradigms, EchoKV utilizes a lightweight network to reconstruct the discarded KV components from a partial subset, exploiting intrinsic inter-layer and intra-layer similarities among attention heads. We further introduce a lightweight two-stage fine-tuning strategy, requiring only a few minutes on a single A100 GPU for Llama-3.1-8B. Experimental results on LongBench and RULER demonstrate that EchoKV offers a competitive quality-cache trade-off across multiple compression ratios and backbone models while preserving the throughput of full-cache inference in short-context scenarios.
Submission history
From: Shiyu Ji [view email]
[v1] Tue, 24 Mar 2026 07:58:42 UTC (276 KB)
[v2] Wed, 13 May 2026 05:25:05 UTC (235 KB)
[v3] Tue, 3 Nov 2026 12:27:35 UTC (550 KB)
