artificialintelbulletin.com

Category: Tutorials

  • StreamDecisionBench: Evaluating Decisions in Force on Evolving Language Streams

    StreamDecisionBench: Evaluating Decisions in Force on Evolving Language Streams

    [Submitted on 29 Sep 2026 (v1), last revised 9 Oct 2026 (this version, v3)]

    View PDF HTML (experimental)

    Abstract:Language models increasingly make real-time decisions in applications that apply the latest answer until a newer one arrives. A late answer can prolong an outdated decision, such as a call recorder still running while a customer reads out card details, an error offline accuracy misses. We make three contributions. First, we release StreamDecisionBench (SDB), a dataset of eight streaming scenarios in four application families, with executable reference decisions derived from public rules. Second, we propose an evaluation protocol and a metric, in-force accuracy: the share of time the applied decision is correct across update intervals of 0.5-8 s. It reflects accuracy and latency jointly, attributing each error to judgment, latency or both. Third, we evaluate fifteen single-model settings, and this attribution separates speed-limited from judgment-limited models: slower, more accurate models lose 42-47% of the time to outdated answers, a fast model 35% to wrong ones. We therefore test hybrids in which a slow model corrects a fast one; with the right pairing and configuration, a hybrid outperforms every single model. However, even the best evaluated system keeps a correct decision in force only about two-thirds of the time, leaving a substantial gap for real-time use.

    Submission history

    From: Jhen-Ke Lin [view email]
    [v1] Tue, 29 Sep 2026 22:15:49 UTC (223 KB)
    [v2] Thu, 1 Oct 2026 14:48:21 UTC (261 KB)
    [v3] Fri, 9 Oct 2026 16:51:55 UTC (328 KB)

  • Learning by Compression: Risk Estimates Attain the Law-of-Large-Numbers Rate

    Learning by Compression: Risk Estimates Attain the Law-of-Large-Numbers Rate

    [Submitted on 9 Oct 2026]

    View PDF HTML (experimental)

    Abstract:Compression schemes provide a general framework for deriving data-dependent risk bounds for learning algorithms. For learning rules based on preferent compression, existing results upper-bound the risk in terms of a sample-dependent complexity $boldsymbol{k}$, defined as the cardinality of the compression augmented by the training examples on which the reconstructed hypothesis is inappropriate. These results, however, yield a margin of order $O(sqrt{ln N/N})$ above $boldsymbol{k}/N$. In this paper, we establish that the logarithmic factor is not intrinsic. We prove that, for every fixed confidence level $1-delta$, the risk is bounded with probability at least $1-delta$ by $$ frac{boldsymbol{k}}{N} + Oleft(frac{1}{sqrt{N}}right), $$ uniformly over all possible values of $boldsymbol{k}$. Thus, risk estimates for learning algorithms based on preferent compression attain the same law-of-large-numbers rate as the empirical estimate of the probability of a fixed event. We also show that the leading term $boldsymbol{k}/N$ is necessary and investigate its relation to the empirical risk. In particular, we identify regimes in which the empirical risk retains the $O(1/sqrt{N})$ convergence rate, as well as regimes in which it converges more slowly or ceases to be consistent, while the complexity-based estimator preserves the $O(1/sqrt{N})$ rate.

    Submission history

    From: Simone Garatti [view email]
    [v1] Fri, 9 Oct 2026 11:44:43 UTC (24 KB)

  • Experience Makes Skillful: Enabling Generalizable Medical Agent Reasoning via Self-Evolving Skill Memory

    Experience Makes Skillful: Enabling Generalizable Medical Agent Reasoning via Self-Evolving Skill Memory

    [Submitted on 8 Jun 2026 (v1), last revised 9 Oct 2026 (this version, v4)]

    View PDF HTML (experimental)

    Abstract:Medical agent systems are increasingly expected to support interactive clinical decision making rather than only static question answering. In such settings, effective agents must reuse prior experience across evolving cases, yet existing memory mechanisms often retain raw historical traces that are redundant, noisy, and difficult to govern. More importantly, they rarely distinguish which memories are truly useful for future reasoning. This limits their ability to accumulate compact and reliable experience for long-horizon clinical reasoning. To close this gap, we propose SkeMex, a post-deployment self-evolution framework that improves medical agents through a skill-based memory without updating model weights. SkeMex distills informative interaction trajectories into structured skills that encode reusable procedural knowledge, and organizes them into a multi-branch repository spanning general, task-specific, and action-level experience. To determine which memories should be reused and retained, SkeMex estimates context-dependent utility from environment feedback and uses it to guide value-aware retrieval and repository governance. A closed-loop “Read–Write–Assess–Govern” lifecycle further supports continual evolution by writing new skills, updating utilities, promoting useful memories, and removing harmful entries. Experiments across diverse clinical tasks show that SkeMex consistently outperforms representative memory-based agents in both offline and online settings. It also generalizes across model backbones and supports transferable skill memory. All data and code will be released publicly.

    Submission history

    From: Haoran Sun [view email]
    [v1] Mon, 8 Jun 2026 11:37:01 UTC (19,046 KB)
    [v2] Wed, 10 Jun 2026 14:46:00 UTC (19,046 KB)
    [v3] Mon, 15 Jun 2026 15:01:33 UTC (19,046 KB)
    [v4] Fri, 9 Oct 2026 07:02:59 UTC (19,057 KB)

  • A Critical Look at Power Sampling when Test Time Sharpening LLMs

    A Critical Look at Power Sampling when Test Time Sharpening LLMs

    [Submitted on 9 Oct 2026]

    View PDF HTML (experimental)

    Abstract:Reinforcement learning with verifiable rewards (RLVR) has substantially improved LLM reasoning, yet growing evidence suggests it mainly performs distribution sharpening: concentrating probability mass on reasoning paths the base model already produces. Recent work therefore seeks RLVR’s gains without needing reward signals, by sampling from the power distribution $pi_alpha propto pi_{text{base}}^alpha$ via Markov chain Monte Carlo or sequential Monte Carlo. We show that these samplers fail to converge to $pi_alpha$ under realistic compute budgets, and that their gains are largely recreated via simple Best-of-$N$ with appropriate grading . This suggests they succeed by selecting high-probability sequences, not by approximating $pi_alpha$, raising a natural question: is the power distribution itself a worthwhile target? We test this hypothesis directly by fine-tuning the base model to match $pi_alpha$, sidestepping sampler convergence issues entirely. Despite approximating the target far more faithfully than existing samplers, this strategy yields gains that are noisy and specific to certain models and sampling tasks. Together, our results suggest that the value of power sampling lies not in faithfully targeting $pi_alpha$, but in the simpler goal of extracting sequences the base model already deems likely.

    Submission history

    From: Abiel Malkani Talwar [view email]
    [v1] Fri, 9 Oct 2026 14:29:28 UTC (235 KB)

  • GeopoliticalQA: Benchmarking Retrieval and Generation over Intelligence Reports

    GeopoliticalQA: Benchmarking Retrieval and Generation over Intelligence Reports

    [Submitted on 8 Oct 2026]

    View PDF HTML (experimental)

    Abstract:Geopolitical and strategic intelligence reports constitute a rich source of expert knowledge and have increasing value in the current political climate, but their heterogeneity and analytical complexity pose challenges for retrieval-augmented question answering. We introduce a benchmark for grounded question answering over 415 open-source geopolitical and strategic intelligence reports, comprising 5,525 document-grounded question-answer pairs spanning factual, inferential, comparative, and risk-oriented questions. We propose a document-aware generation pipeline that profiles reports before generating questions and validate dataset quality through expert annotation and an LLM-as-judge protocol. Using the benchmark, we evaluate 15 retrieval systems and six open-weight LLMs for RAG. We find that models perform similarly on document-level retrieval, while at the harder chunk level retrieval quality is not driven by embedding-model scale: bge-m3, a mid-size open encoder matches an order of magnitude larger ones and outperforms commercial embedding APIs. In contrast, answer generation remains challenging even given the gold evidence, particularly for inferential and risk-oriented questions.

    Submission history

    From: Veronika Solopova [view email]
    [v1] Thu, 8 Oct 2026 18:35:13 UTC (67 KB)

  • Equality, Equity, and Causality in Fairness Research: A Commentary on Cheng (2026)

    Equality, Equity, and Causality in Fairness Research: A Commentary on Cheng (2026)

    [Submitted on 20 Jul 2026 (v1), last revised 9 Oct 2026 (this version, v2)]

    View PDF HTML (experimental)

    Abstract:This is an invited commentary on the Psychometrika focus article “Fairness Issues and Evaluation in Psychometrics and AI/ML: What Can We Learn from Each Field?” by Ying Cheng (2026, doi:https://doi.org/10.1017/psy.2026.10110). Cheng offers a systematic comparison between long-standing test fairness and modern algorithmic fairness. Her mapping of the entire testing workflow onto the AI/ML fairness paradigm, rather than only the final selection stage, is a crucial contribution to interdisciplinary fairness research. This commentary extends her discussion by examining two conceptual issues: the distinction between equality and equity, and the role of causality in fairness research. The focus article and this commentary point to directions for future fairness research across the psychometrics and AI/ML communities.

    Submission history

    From: Youmi Suk [view email]
    [v1] Mon, 20 Jul 2026 08:30:06 UTC (14 KB)
    [v2] Fri, 9 Oct 2026 12:26:26 UTC (14 KB)

  • perturber: A Stateless Service for Semantics-Preserving Text Perturbations

    perturber: A Stateless Service for Semantics-Preserving Text Perturbations

    [Submitted on 8 Oct 2026]

    View PDF HTML (experimental)

    Abstract:Large language model (LLM) evaluations on clean benchmarks overstate model robustness: real inputs arrive perturbed, and these meaning-preserving variants routinely move model answers. Studying this systematically requires applying perturbations one at a time, reproducibly, and identically wherever they run, yet existing tools couple them to a benchmarking stack rather than packaging them as a reusable transform layer. We present perturber, a stateless service and Python library that applies exactly one semantics-preserving perturbation per call, drawn from a first-party catalog of 68 perturbations across 11 families. It turns on a single primitive, with composition left to the caller chaining calls, so the service and the library produce identical output by construction. We evaluate the tool itself, its correctness, reproducibility, and cost, rather than any model’s robustness, measuring meaning-preservation per perturbation rather than assuming it. The demonstration adds a live playground, an introspectable catalog for benchmark selection, and a one-command reproducible evaluation.

    Submission history

    From: Khoi-Nguyen Tran [view email]
    [v1] Thu, 8 Oct 2026 19:44:05 UTC (153 KB)

  • The Predictability Bias of Joint-Embedding Predictive Learning in Medical Imaging

    The Predictability Bias of Joint-Embedding Predictive Learning in Medical Imaging

    [Submitted on 9 Oct 2026]

    View PDF HTML (experimental)

    Abstract:Joint-embedding predictive architectures (JEPAs) learn by predicting teacher representations of masked regions from visible context. Point prediction summarizes the conditional target distribution and cannot reproduce variation that the context does not resolve. We call this constraint predictability bias. In computed tomography (CT), recurrent anatomy is predictable from context, whereas the sporadic, localized pathology that determines clinical interpretation often is not. We introduce TAP-JEPA, a ViT-B/3D trained on 113,611 CT volumes that is competitive with state-of-the-art encoders (COLIPRI, Merlin, VoxelFM) in scan-level classification. We then audit the learned predictive pathway by replaying the exact masking, teacher, and predictor computations used during training. Prediction lowers the mean Skill (prevalence-normalized average precision) from 0.388 to 0.271 for pathology, but only from 0.921 to 0.873 for anatomy. Along the teacher direction, separating pathology from matched control regions in the same organ, prediction moves pathology 34.7% toward the controls and controls 13.6% toward pathology, with larger shifts farther from visible context. Replacing visible matched-control representations increases surrounding-target prediction loss more than fully replacing pathology representations across four pathology categories. Downstream, removing all final-layer pathology tokens reduces the decision margin by at most 0.45% in four of seven scan-level probes. Together, these results show that contextual predictability is not a sufficient proxy for clinical relevance in medical representation learning. Training objectives should therefore retain diagnostic features regardless of their predictability from surrounding anatomy.

    Submission history

    From: Fabian Drexel [view email]
    [v1] Fri, 9 Oct 2026 08:53:13 UTC (39,204 KB)

  • Reflective Prompt Optimization of a Deep-Research Agent: A Topic-Disjoint, Multi-Seed Protocol for Measuring Transferable Gains and Prompt-Level Overfitting

    Reflective Prompt Optimization of a Deep-Research Agent: A Topic-Disjoint, Multi-Seed Protocol for Measuring Transferable Gains and Prompt-Level Overfitting

    [Submitted on 8 Oct 2026]

    View PDF HTML (experimental)

    Abstract:Prompt optimization is an attractive alternative to reinforcement learning for adapting language-model agents, since it is cheap and produces human-readable artifacts. On factual tasks, however, it carries a failure mode that scalar-reward reinforcement learning does not. An optimizer that reads the training examples can write their answers into the prompt, so an in-distribution gain may be memorization rather than transferable behaviour. We apply reflective prompt evolution (GEPA) to the two system prompts of a multi-agent deep-research agent and evaluate it under a topic-disjoint split in which no topic is shared across train, validation, and test. Held-out citation reward rises by 0.079, from 0.702 to 0.781, with a bootstrap 95% confidence interval of [0.051, 0.107], and the gain replicates across three seeds (mean 0.107). Nearly half of the winning prompt is a memorized answer key, yet a controlled ablation shows this content earns no reward even on the exact training questions it encodes. The large validation-minus-held-out gap is therefore small-sample selection under a noisy judge, not memorization. Varying the verifier isolates what makes the gain transfer. Decomposed per-dimension feedback is the causal ingredient, and collapsing it to a single scalar reward makes held-out transfer negative. The gain persists under an out-of-family judge and is largely reproduced by a hand-written version of the discovered recipe. Credible evaluation of prompt optimizers therefore requires held-out topics, multiple seeds, and verifier ablations. We release the candidate prompts, lineage diffs, figures, tables, and analysis.

    Submission history

    From: Sahil Sen [view email]
    [v1] Thu, 8 Oct 2026 18:23:23 UTC (77 KB)

  • The Fourth Quadrant: A Stylized View of Benign Misfitting

    The Fourth Quadrant: A Stylized View of Benign Misfitting

    [Submitted on 2 Aug 2026 (v1), last revised 8 Oct 2026 (this version, v2)]

    View PDF HTML (experimental)

    Abstract:Training error is what we can observe on a training set; test error is the quantity we care about. We study squared-error linear regression in a deterministic $(d+1)$-dimensional single-spike model. Each training vector has spike coordinate $sqrt{gamma}$, $gamma>1$, and equal-norm, mutually orthogonal nuisance components. Training labels are $1$. Fresh test points are centered Gaussian with covariance $operatorname{diag}(gamma,1,ldots,1)$ and noise-free label $x[1]/sqrt{gamma}$. We compute the exact train–test tradeoff for linear predictors in the training span. When the spike direction is unknown, allowing all linear predictors does not improve this tradeoff, on average or in the worst case over directions consistent with the data. There is a range of training-set sizes where every span predictor that generalizes well must fit the training data worse than the zero predictor. We call this benign misfitting, or the fourth quadrant. At its sharpest, test error tends to zero while training error diverges, with both zero-predictor errors normalized to one. The best span predictor begins to generalize when $ngg d/gamma^2$, while interpolation needs $ngg d/gamma$. In the intervening window $d/gamma^2ll nll d/gamma$, useful span prediction lies beyond interpolation: training predictions overshoot the labels. One-pass stochastic gradient descent with a large constant learning rate attains small test error throughout this window, matching the best span predictor up to a logarithmic factor, while leaving large empirical training error. Finally, the nuisance component responsible for the training misfit also controls adversarial sensitivity.

    Submission history

    From: Anant Sahai [view email]
    [v1] Sun, 2 Aug 2026 06:27:55 UTC (136 KB)
    [v2] Thu, 8 Oct 2026 23:13:07 UTC (202 KB)