[Submitted on 12 Nov 2026]
Abstract:Memory decides which LLMs an edge device runs, and how fast: capacity on a $4$ GB Jetson Nano, bandwidth on an $8$ GB Jetson Orin Nano Super. Deployed formats (GPTQ, AWQ, $texttt{this http URL}$’s K- and I-quants) spend equal bits on every integer code of a matrix, although the codes are far from uniform. $texttt{LazyWeights}$ keeps these formats but tunes GPTQ so the codes compress, Huffman-codes them losslessly in interleaved lanes, and decodes them on demand in a small unit between DRAM and L2, so GPU kernels run unchanged and DRAM moves only coded bytes. At $1%$ and $3.5%$ perplexity budgets, coded Phi-3-mini, Mistral-7B and SmolLM-1.7B are $5.7-31.4%$ smaller than the best $texttt{this http URL}$ format. A bit-exact $0.21$ mm$^2$ ASIC decoder unit ($1$ GHz, $7$ nm) cuts Orin latency and energy per token by up to $36%$ and $38%$ ($54%$ and $44%$ with prompt-lookup speculation). On the Nano, $texttt{LazyWeights}$ needs $58-64%$ less time and $53-57%$ less energy per token than CPU $texttt{this http URL}$, and runs a streamed Mistral-7B 6.6$times$ faster at $3.89$ bits per weight. A Huffman decoder beats an rANS decoder in rate and area at equal throughput; the ASIC decoder unit runs bit-exact on a Zynq UltraScale+ FPGA (AMD ZCU102).
Submission history
From: Arnab Sanyal [view email]
[v1] Thu, 12 Nov 2026 17:03:40 UTC (411 KB)
