artificialintelbulletin.com

Tag: cs.LG

  • Adaptive DHiSparse: Speculative Decoding over a Load-Adaptive KV Hierarchy

    Adaptive DHiSparse: Speculative Decoding over a Load-Adaptive KV Hierarchy

    [Submitted on 9 Oct 2026]

    View PDF HTML (experimental)

    Abstract:Long-context serving runs out of KV memory before the hardware reaches its useful throughput. HiSparse stores history in host memory and fetches only the selected entries, but it uses one cache budget at every load and does not handle draft-model speculative decoding such as DSpark and DFlash2. We present Adaptive DHiSparse, which supports speculative decoding over hierarchical sparse KV and adapts residency to load. It protects the required working set, retains additional history while GPU memory is spare, and recovers the rest through exact fused retrieval. DSpark and DFlash2 keep their proposal and acceptance rules: queries share missing-row retrieval, draft KV uses a window-bounded ring, and verification staging is reused across layer groups. At 100K context and concurrency 128, GLM-5.3 with DFlash2 delivers 793 output tokens/s per decode GPU, 1.45$times$ HiSparse and 1.27$times$ non-speculative adaptive serving. Held to the same latency bounds as HiSparse, it sustains up to 4$times$ the concurrency and 2.3 to 3.7$times$ the output, depending on the per-token bound. On DeepSeek-V4.1-Flash at 500K context and concurrency 512, Adaptive DHiSparse with DSpark delivers 1.85$times$ the output of resident serving with DSpark. In a load step, optional residency reduces host fetches from 14.2% to 1.4% of selected rows at low load and releases memory under pressure.

    Submission history

    From: Naji Alazhar [view email]
    [v1] Fri, 9 Oct 2026 15:02:35 UTC (309 KB)

  • The Predictability Bias of Joint-Embedding Predictive Learning in Medical Imaging

    The Predictability Bias of Joint-Embedding Predictive Learning in Medical Imaging

    [Submitted on 9 Oct 2026]

    View PDF HTML (experimental)

    Abstract:Joint-embedding predictive architectures (JEPAs) learn by predicting teacher representations of masked regions from visible context. Point prediction summarizes the conditional target distribution and cannot reproduce variation that the context does not resolve. We call this constraint predictability bias. In computed tomography (CT), recurrent anatomy is predictable from context, whereas the sporadic, localized pathology that determines clinical interpretation often is not. We introduce TAP-JEPA, a ViT-B/3D trained on 113,611 CT volumes that is competitive with state-of-the-art encoders (COLIPRI, Merlin, VoxelFM) in scan-level classification. We then audit the learned predictive pathway by replaying the exact masking, teacher, and predictor computations used during training. Prediction lowers the mean Skill (prevalence-normalized average precision) from 0.388 to 0.271 for pathology, but only from 0.921 to 0.873 for anatomy. Along the teacher direction, separating pathology from matched control regions in the same organ, prediction moves pathology 34.7% toward the controls and controls 13.6% toward pathology, with larger shifts farther from visible context. Replacing visible matched-control representations increases surrounding-target prediction loss more than fully replacing pathology representations across four pathology categories. Downstream, removing all final-layer pathology tokens reduces the decision margin by at most 0.45% in four of seven scan-level probes. Together, these results show that contextual predictability is not a sufficient proxy for clinical relevance in medical representation learning. Training objectives should therefore retain diagnostic features regardless of their predictability from surrounding anatomy.

    Submission history

    From: Fabian Drexel [view email]
    [v1] Fri, 9 Oct 2026 08:53:13 UTC (39,204 KB)

  • Quadratic Characterizations for Reachability Analysis of Neural Networks

    Quadratic Characterizations for Reachability Analysis of Neural Networks

    [Submitted on 19 May 2026 (v1), last revised 9 Oct 2026 (this version, v2)]

    View PDF HTML (experimental)

    Abstract:Quadratic constraints (QCs) are widely used to characterize nonlinearities and uncertainties, but generic analytical characterizations can be conservative on bounded domains. This paper develops a framework for constructing verified quadratic characterizations of scalar relations in the two-dimensional real plane. Candidate quadratic inequalities are locally generated by solving convex quadratic programs using samples from the relation and exterior sample points. They are then verified globally using sum-of-squares certificates over an exact semialgebraic description or, in the case of nonpolynomial relations, over relaxed polynomial descriptions. The resulting verified constraints define a sound overapproximation of the scalar relations over the considered domains. These constraints are directly compatible with existing analysis frameworks based on QCs and pointwise integral quadratic constraints (IQCs) for static nonlinearities and uncertainties, and they can also be embedded in QC-based semidefinite programs for reachability and safety analysis of feedforward neural networks. For smooth activations such as $tanh$, the method yields domain-dependent quadratic characterizations that constitute an alternative to generic sector- or slope-based descriptions. For ReLU networks, we give methods to reduce conservatism in QC-based reachability analysis of feedforward networks by exploiting dependencies between neurons and tighter local bounds. Numerical examples demonstrate improved reachability results for smooth activations, reduced conservatism for ReLU networks, and applicability beyond neural networks through an example involving saturation.

    Submission history

    From: Elias Khalife [view email]
    [v1] Tue, 19 May 2026 20:45:31 UTC (735 KB)
    [v2] Fri, 9 Oct 2026 04:57:49 UTC (738 KB)

  • Machine Learning and Time-Series Foundation Models for Day-Ahead Electricity Forecasting in a University Building

    Machine Learning and Time-Series Foundation Models for Day-Ahead Electricity Forecasting in a University Building

    [Submitted on 8 Oct 2026]

    View PDF HTML (experimental)

    Abstract:Day-ahead electricity forecasts can help building operators schedule heating, ventilation, and other services around anticipated demand. Choosing a forecasting model also requires balancing accuracy with its forecast-time inputs, local training, and computational requirements.
    We benchmark statistical, machine-learning, recurrent neural-network, and pretrained TimesFM models for a university building. We use chronological validation and evaluate all configurations on the same 57 test days, comparing accuracy across workdays, weekends, and holidays alongside local fitting and forecasting time. Weather covariates use retrospective observations.
    TimesFM 3.0 with covariates achieves the lowest overall errors, reducing mean absolute error by approximately 43% relative to weekly seasonal-naive forecasting. XGBoost remains competitive, although paired daily comparisons do not establish a clear mean-absolute-error advantage for TimesFM. TimesFM also performs strongly using only past electricity consumption. Adding covariates improves its holiday forecasts but slightly worsens workday and weekend forecasts, leaving overall mean absolute error almost unchanged. TimesFM requires no building-specific weight training, while XGBoost offers rapid local fitting and inference. The evaluated LSTM takes longer to fit without outperforming either approach.
    An exploratory historical evaluation reveals seasonal variation, while a separate weather-source comparison produces only small changes in the selected TimesFM configuration’s accuracy. These results support pretrained TimesFM as a strong initial forecasting baseline for this building, with XGBoost as a competitive locally fitted alternative. Covariates are particularly useful when improved holiday forecasts justify the additional input preparation.

    Submission history

    From: Amey Vasulkar [view email]
    [v1] Thu, 8 Oct 2026 19:00:09 UTC (2,871 KB)

  • The Fourth Quadrant: A Stylized View of Benign Misfitting

    The Fourth Quadrant: A Stylized View of Benign Misfitting

    [Submitted on 2 Aug 2026 (v1), last revised 8 Oct 2026 (this version, v2)]

    View PDF HTML (experimental)

    Abstract:Training error is what we can observe on a training set; test error is the quantity we care about. We study squared-error linear regression in a deterministic $(d+1)$-dimensional single-spike model. Each training vector has spike coordinate $sqrt{gamma}$, $gamma>1$, and equal-norm, mutually orthogonal nuisance components. Training labels are $1$. Fresh test points are centered Gaussian with covariance $operatorname{diag}(gamma,1,ldots,1)$ and noise-free label $x[1]/sqrt{gamma}$. We compute the exact train–test tradeoff for linear predictors in the training span. When the spike direction is unknown, allowing all linear predictors does not improve this tradeoff, on average or in the worst case over directions consistent with the data. There is a range of training-set sizes where every span predictor that generalizes well must fit the training data worse than the zero predictor. We call this benign misfitting, or the fourth quadrant. At its sharpest, test error tends to zero while training error diverges, with both zero-predictor errors normalized to one. The best span predictor begins to generalize when $ngg d/gamma^2$, while interpolation needs $ngg d/gamma$. In the intervening window $d/gamma^2ll nll d/gamma$, useful span prediction lies beyond interpolation: training predictions overshoot the labels. One-pass stochastic gradient descent with a large constant learning rate attains small test error throughout this window, matching the best span predictor up to a logarithmic factor, while leaving large empirical training error. Finally, the nuisance component responsible for the training misfit also controls adversarial sensitivity.

    Submission history

    From: Anant Sahai [view email]
    [v1] Sun, 2 Aug 2026 06:27:55 UTC (136 KB)
    [v2] Thu, 8 Oct 2026 23:13:07 UTC (202 KB)

  • Transformer-Based Masked Diffusion Models Provably Learn Distributions with Latent Structure

    Transformer-Based Masked Diffusion Models Provably Learn Distributions with Latent Structure

    [Submitted on 8 Oct 2026]

    View PDF HTML (experimental)

    Abstract:Masked diffusion models (MDMs) generate discrete data by iteratively unmasking multiple tokens in parallel using a learned mask predictor. Their output carries two errors: parallelization error due to parallel sampling, and estimation error from the mask predictor learned with finite samples. Existing theory treats these errors separately, and the available statistical bounds typically scale with the number $V^L$ of length-$L$ sequences over a vocabulary of size $V$, suffering from the curse of dimensionality. In this work, we establish end-to-end statistical guarantees showing that transformer-based MDMs learn structured distributions with a sample complexity governed by their latent structure. Specifically, we model this structure through mixtures of hidden Markov trees, which capture latent dependencies within components and heterogeneity across components. When the data distribution is well approximated by such a mixture, a transformer-based MDM with parallel unmasking schedules achieves $varepsilon$-accurate sampling in KL divergence with $widetilde{O}(LK_{mathrm{par}}/varepsilon)$ training samples, where $K_{mathrm{par}}$ is polynomial in $L$, $V$, the number of hidden states, and the number of mixture components. The transformer network and sampling procedure require no knowledge of the tree topologies or component parameters, thereby adapting to the unknown latent structure automatically. Overall, our results provide theoretical justification for the empirical effectiveness of MDMs on real-world data.

    Submission history

    From: Jingda Wu [view email]
    [v1] Thu, 8 Oct 2026 17:54:49 UTC (90 KB)

  • TernaryLLM: Ternarized Large Language Model

    TernaryLLM: Ternarized Large Language Model

    [Submitted on 11 Jun 2024 (v1), last revised 9 Oct 2026 (this version, v2)]

    View PDF HTML (experimental)

    Abstract:Large language models (LLMs) have achieved remarkable performance on Natural Language Processing (NLP) tasks, but they are hindered by high computational costs and memory requirements. Ternarization, an extreme form of quantization, offers a solution by reducing memory usage and enabling energy-efficient floating-point additions. However, applying ternarization to LLMs faces challenges stemming from outliers in both weights and activations. In this work, observing asymmetric outliers and non-zero means in weights, we introduce Dual Learnable Ternarization (DLT), which enables both scales and shifts to be learnable. We also propose Outlier-Friendly Feature Knowledge Distillation (OFF) to recover the information lost in extremely low-bit quantization. The proposed OFF can incorporate semantic information and is insensitive to outliers. At the core of OFF is maximizing the mutual information between features in ternarized and floating-point models using cosine similarity. Extensive experiments demonstrate that our TernaryLLM surpasses previous low-bit quantization methods on the standard text generation and zero-shot benchmarks for different LLM families. Specifically, for one of the most powerful open-source models, LLaMA-3, our approach (W1.58A16) outperforms the previous state-of-the-art method (W2A16) by 5.8 in terms of perplexity on C4 and by 8.2% in terms of average accuracy on zero-shot tasks.

    Submission history

    From: Chen Tianqi [view email]
    [v1] Tue, 11 Jun 2024 11:40:12 UTC (674 KB)
    [v2] Fri, 9 Oct 2026 06:58:39 UTC (675 KB)

  • Infinitesimal Operator World Models: Finite-Sample Guarantees for Offline Reinforcement Learning

    Infinitesimal Operator World Models: Finite-Sample Guarantees for Offline Reinforcement Learning

    [Submitted on 13 Nov 2025 (v1), last revised 9 Oct 2026 (this version, v2)]

    View PDF HTML (experimental)

    Abstract:Continuous-time stochastic processes underlie many natural and engineered systems. In healthcare, autonomous driving, and industrial control, direct interaction with the environment is often unsafe or impractical, motivating offline reinforcement learning from historical data. However, there is limited statistical understanding of the approximation errors inherent in learning policies from offline datasets. We address this by linking reinforcement learning to the Hamilton-Jacobi-Bellman equation and proposing an operator-theoretic algorithm based on a simple dynamic programming recursion. Specifically, we represent our world model in terms of the infinitesimal generator of controlled diffusion processes learned in a reproducing kernel Hilbert space. By integrating statistical learning methods and operator theory, we establish global convergence of the value function under sufficiently small generator-approximation error and derive explicit finite-sample guarantees with bounds tied directly to system properties such as smoothness and stability. Our theoretical and proof-of-concept results indicate that operator-based approaches may hold promise for solving offline reinforcement learning using continuous-time optimal control.

    Submission history

    From: Nicolas Hoischen [view email]
    [v1] Thu, 13 Nov 2025 14:58:30 UTC (1,427 KB)
    [v2] Fri, 9 Oct 2026 13:08:08 UTC (2,717 KB)

  • Measuring and Mitigating Solution Mode Collapse in RLVR

    Measuring and Mitigating Solution Mode Collapse in RLVR

    [Submitted on 8 Oct 2026 (v1), last revised 9 Oct 2026 (this version, v2)]

    View PDF HTML (experimental)

    Abstract:A language model (LM) can usually answer the same question in more than one way, but reinforcement learning with verifiable rewards (RLVR) is indifferent to which correct answer a model produces. A solution will earn the same reward whether it is the thousandth copy of a familiar answer or one the model has never produced before. Yet, there is potential value in having the model retain multiple correct solutions as it is trained. For instance, multiple modes may give users a choice and provide problem-solving strategies that improve overall model performance. Here, we introduce ModeBench, a benchmark of multi-solution tasks in which the verifier returns both correctness and mode discovered. We then use ModeBench to measure how solution diversity changes under RLVR post-training. We find that RLVR post-training concentrates probability onto fewer correct modes even as accuracy holds or improves, and moreover, that frontier models are already highly concentrated. We then introduce our solution, Re:Max, which stores one verified example per discovered mode in a replay buffer and trains on those stored modes uniformly. A solution found once is, therefore, practiced as often as one found repeatedly. Across three model scales, two RL objectives, and harder task constructions, replay improves both how often a policy succeeds and how many different ways it can succeed.

    Submission history

    From: Liv D’Aliberti [view email]
    [v1] Thu, 8 Oct 2026 01:25:16 UTC (744 KB)
    [v2] Fri, 9 Oct 2026 02:22:30 UTC (744 KB)

  • SynAct: Synergistic Active Feedback Learning for LLM Agents

    SynAct: Synergistic Active Feedback Learning for LLM Agents

    [Submitted on 8 Oct 2026]

    View PDF HTML (experimental)

    Abstract:LLM agents executing multi-step workflows fail in two fundamentally different ways: recoverable slips, which further attempts can repair, and knowledge gaps, which retrying cannot resolve. Expert feedback can address the latter, but expert attention is scarce. The key challenge is therefore not whether to ask for feedback, but where feedback can change the outcome. Existing approaches either avoid expert feedback, allowing agents to reinforce their own errors, or allocate feedback at the task level without identifying consequential decisions. We present SynAct, an active learning framework that treats expert attention as the binding constraint and allocates feedback at the level of individual decisions. SynAct ranks decisions by the mutual information between the selected action and the terminal outcome across repeated rollouts, queries an expert at the highest-value decisions, and distills the resulting corrections into transferable rules that the agent can reuse without expert access at deployment. We theoretically characterize what this signal identifies under finite repeated sampling: with k rollouts and at least m observations per action, outcome-constant decisions have zero mutual information, while outcome-separating decisions have mutual information at least H2(m/k) bits, yielding exact separation for any threshold t in (0,H2(m/k)). This also explains why success-based acquisition can be blind when all rollouts fail: success is constant even when failed runs reach different outcomes. Across four agent benchmarks and four models spanning three families, SynAct improves held-out accuracy in all 16 evaluated settings, by +11.7pp on average. The same signal transfers to inference-time routing, where SynAct outperforms budget-matched cascade, uncertainty, and expert-judge routers in 15 of 16 settings, while a single targeted correction outperforms seven additional unaimed retries.

    Submission history

    From: Ning Liu [view email]
    [v1] Thu, 8 Oct 2026 19:20:10 UTC (642 KB)