artificialintelbulletin.com

Tag: cs.LG

  • Market Impact Estimation via Causal Foundation Models

    Market Impact Estimation via Causal Foundation Models

    [Submitted on 8 Oct 2026]

    View PDF HTML (experimental)

    Abstract:When a trader executes a large order, the relevant question is not emph{what will the price be?} but emph{what would the price have been had I not traded?} Market impact is thus a causal quantity, yet classical models (Almgren–Chriss, propagators) regress price on flow, which is biased whenever informed flow moves prices through information as well as trading. We present a proof-of-concept emph{Causal Market Simulator}: a continuous-time causal prior-data fitted network (PFN) trained on a synthetic market prior in which a hidden informational flow confounds the flow–price relation by construction, with confounding strength an exact per-episode knob. On this prior, the naive estimator’s bias follows a closed form we confirm empirically (up to 68% overestimation), while the PFN tracks the unbiased true-parameter oracle at high confounding. Its emph{do-ablated twin}, the same model with the intervention inputs removed, collapses onto the pre-window mean. Zero-shot, the model responds at the ground-truth scale to metaorders in an ABIDES limit-order book ($+0.38$ pre-onset standard deviations against seed-replay counterfactuals), and its intervention response replicates on about 2,300 detected flow bursts in BTC/ETH/SOL perpetuals and 3,800 on Nasdaq equities, evidence consistent with, though not proof of, causal transfer. Off-prior the response degrades gracefully, and split-conformal recalibration repairs interval calibration. Transfer hinges on encoding the metaorder as added flow rather than a clamped price level, not on more training. A recurrent state-space variant closes the remaining shape gap: its impact accrues over execution duration where the transformer’s stays flat, raising the correlation with measured metaorder impact from $+0.68$ to between $+0.92$ and $+0.96$, at the cost of weaker on-prior deconfounding that traces to the training recipe rather than to recurrence.

    Submission history

    From: Dennis Thumm [view email]
    [v1] Thu, 8 Oct 2026 13:29:39 UTC (182 KB)

  • Smoothing the Top-k Exposure Boundary for Sparse Mixture-of-Experts

    Smoothing the Top-k Exposure Boundary for Sparse Mixture-of-Experts

    [Submitted on 8 Oct 2026 (v1), last revised 9 Oct 2026 (this version, v2)]

    View PDF HTML (experimental)

    Abstract:Sparse Mixture-of-Experts models scale parameter capacity efficiently while maintaining a fixed compute budget per token. However, traditional training paradigms enforce a static choice of top-$k$ experts, which converts a continuous routing distribution into a rigid step function. This constraint introduces a brittle boundary where highly competitive experts are arbitrarily separated into full-supervision and zero-feedback zones based on minor score fluctuations. To address this issue, we propose Elastic Expert Routing, which stochastically samples the active expert budget from a localized discrete distribution centered at $k$. Over multiple training iterations, this mechanism softens the sharp threshold into a gradual probability distribution. Because the sampling neighborhood remains symmetric, this approach matches the expected computational cost of deterministic training, while preserving the inference budget. During supervised fine-tuning, elastic routing improves downstream macro-averages on OLMoE-1B-7B and Qwen3-30B-A3B by $+0.84$ and $+2.02$ points, respectively. In addition, in from-scratch pretraining, it reduces validation LM loss by up to $0.04$ and outperforms the static top-$k$ baseline by $1.6$ points on average across downstream tasks. Comparisons on Qwen3 fine-tuning and from-scratch pretraining further show higher average scores than Noisy top-$k$, expert FFN dropout, and DefaultMoE.

    Submission history

    From: Yunkai Chai [view email]
    [v1] Thu, 8 Oct 2026 09:31:19 UTC (157 KB)
    [v2] Fri, 9 Oct 2026 09:32:21 UTC (159 KB)

  • Bayesian dynamic scheduling of multipurpose batch processes under incomplete look-ahead information

    Bayesian dynamic scheduling of multipurpose batch processes under incomplete look-ahead information

    [Submitted on 30 Nov 2025 (v1), last revised 9 Oct 2026 (this version, v2)]

    View PDF HTML (experimental)

    Abstract:Multipurpose batch processes become increasingly popular in manufacturing industries since they adapt to low-volume, high-value products and shifting demands. These processes often operate in a dynamic environment, which faces disturbances such as processing delays and demand changes. To minimise long-term cost and system nervousness (i.e., disruptive changes to schedules), schedulers must design rescheduling strategies to address such disturbances effectively. Existing methods often assume complete look-ahead information over the scheduling horizon. This assumption contrasts with realistic situations where schedulers can only access incomplete look-ahead information. Sticking with existing methods may lead to suboptimal long-term costs and high-level system nervousness. In this work we propose a Bayesian dynamic scheduling method. Our method relies on learning a Bayesian Network from the probability distribution of disturbances. Specifically, the Bayesian Network represents how likely each operation will be impacted by disturbances. During the online execution, when new disturbances become observed, this method updates the posterior distribution and therefore guides the rescheduling strategy. We compare our method with the existing periodic rescheduling strategy (which generates new schedules from scratch at fixed intervals) on four benchmark problems. Computational results show that our method achieves statistically better long-term costs and system nervousness. In the theoretical aspect, we prove that if disturbances are mutually independent, the impact-quantifying variables inherently satisfy the independence assumptions required by Bayesian Networks. As an implication, practitioners can extend the method to other scheduling problems (such as job shop scheduling and continuous processes), given that they define the problem-specific dependencies between operations.

    Submission history

    From: Taicheng Zheng [view email]
    [v1] Sun, 30 Nov 2025 21:27:52 UTC (1,350 KB)
    [v2] Fri, 9 Oct 2026 09:42:06 UTC (2,695 KB)

  • On the Rate of Convergence of GD in Non-linear Neural Networks: An Adversarial Robustness Perspective

    On the Rate of Convergence of GD in Non-linear Neural Networks: An Adversarial Robustness Perspective

    [Submitted on 2 Mar 2026 (v1), last revised 8 Oct 2026 (this version, v2)]

    View PDF

    Abstract:We study the convergence dynamics of Gradient Descent (GD) in a minimal binary classification setting, consisting of a two-neuron ReLU network and two training instances. We prove that even under these strong simplifying assumptions, while GD successfully converges to an optimal robustness margin, effectively maximizing the distance between the decision boundary and the training points, this convergence occurs at a prohibitively slow rate, scaling strictly as $Theta(1/ln(t))$. To the best of our knowledge, this establishes the first explicit lower bound on the convergence rate of the robustness margin in a non-linear model. Through empirical simulations, we further demonstrate that this inherent failure mode is pervasive, exhibiting the exact same tight convergence rate across multiple natural network initializations. Our theoretical guarantees are derived via a rigorous analysis of the GD trajectories across the distinct activation patterns of the model. Specifically, we develop tight control over the system’s dynamics to bound the trajectory of the decision boundary, overcoming the primary technical challenge introduced by the non-linear nature of the architecture.

    Submission history

    From: Guy Smorodinsky [view email]
    [v1] Mon, 2 Mar 2026 17:13:33 UTC (38 KB)
    [v2] Thu, 8 Oct 2026 18:50:56 UTC (38 KB)

  • Sample-Adaptive Zeroth-Order Post-Training

    Sample-Adaptive Zeroth-Order Post-Training

    [Submitted on 8 Oct 2026]

    View PDF

    Abstract:Zeroth-order (ZO) optimization offers a memory-efficient alternative to backpropagation for fine-tuning large language models. However, existing ZO methods typically sample training examples uniformly, overlooking their evolving importance and leaving avoidable estimation variance. To address this limitation, we propose Sample-Adaptive Zeroth-Order Optimization (SAZO), which learns sample importance online and adaptively allocates queries to accelerate convergence. SAZO extracts gradient-norm information from existing directional differences to estimate sample importance without access to true gradients. Furthermore, to track evolving importance from sparse minibatch feedback, SAZO maintains per-example exponential moving averages that aggregate observations across updates. The method requires no additional model evaluations and only $O(N)$ additional scalar state compared with MeZO. We prove that SAZO asymptotically matches the leading convergence term of the minimum-variance sampling oracle, and achieves a strictly faster convergence rate than standard methods. Extensive experiments across four model families demonstrate improved average downstream performance and up to $6times$ faster convergence to the same training loss in terms of query count.

    Submission history

    From: Haiyuan Wan [view email]
    [v1] Thu, 8 Oct 2026 18:00:37 UTC (305 KB)

  • Kronecker Product Attention: Memory-Efficient KV Compression via Structured Factorization

    Kronecker Product Attention: Memory-Efficient KV Compression via Structured Factorization

    [Submitted on 8 Oct 2026]

    View PDF HTML (experimental)

    Abstract:Efficient autoregressive generation is increasingly limited by the memory footprint of the key-value (KV) cache, which grows linearly with sequence length, batch size, and model depth. Existing approaches such as Multi-Query Attention (MQA), Grouped-Query Attention (GQA), Multi-Head Latent Attention (MLA), and Tensor Product Attention (TPA) reduce this cost through different trade-offs, but often sacrifice expressivity, require additional positional-encoding machinery, or impose restrictive low-rank structure. We propose Kronecker Product Attention (KPA), which represents key and value activations as sums of Kronecker products of compact, token-dependent matrix factors. KPA strictly generalizes TPA: rank-one vector-outer-product factorization arises as a degenerate Kronecker split, while non-degenerate matrix factors can represent richer block-structured patterns. We further show that a structured rotary embedding operator matched to the Kronecker representation preserves the relative-position property of RoPE while rotating only the compact factors, removing the need for a decoupled RoPE branch as in MLA. Empirically, KPA matches or exceeds TPA, MHA, MQA, GQA, and MLA across four model scales and nine downstream benchmarks under identical training conditions, while achieving up to 10x KV cache compression. The Kronecker head split is a tuneable hyperparameter that allows practitioners to navigate the compression-accuracy trade-off at design time without architectural changes.

    Submission history

    From: Bahareh Nikpour [view email]
    [v1] Thu, 8 Oct 2026 19:23:56 UTC (227 KB)

  • GlucoPRISM: Protocol-Supervised Pretraining for Cross-Cohort Glucose Transfer

    GlucoPRISM: Protocol-Supervised Pretraining for Cross-Cohort Glucose Transfer

    [Submitted on 8 Oct 2026]

    View PDF HTML (experimental)

    Abstract:Continuous glucose monitoring (CGM) foundation models encode each day as one vector that mixes who the person is, what happened that day and how the sensor measured it, so measurement differences travel with the representation into new cohorts. We ask whether pretraining can place the measurement component that harms transfer in known coordinates, so that deployment drops them with a fixed projection and no target data. GlucoPRISM supervises a GlucoFM backbone with response contracts, rules for how each part of the representation should change when only the day or only the sensor changes: a 64-dimensional Trait band should change with neither, a 48-dimensional State band only across days, and a 16-dimensional Sensor band should track the device. Synthetic sensor views calibrated on paired Dexcom and Libre recordings, different-day pairs and glucose-summary targets supply this supervision without clinical labels. Across twelve cross-cohort directions, dropping the Sensor band from the same frozen encoder raises mean transfer AUROC from 68.75 to 73.57, against 69.3 for GlucoFM. The gain has two parts. Into CGMacros, the one target cohort recorded with two sensor types, filtering adds 11 to 19 points, and this holds when the generator is recalibrated on half of the participants and scored on the other half. On the other eight directions filtering is neutral and the encoder itself exceeds GlucoFM by 3.5 points. The device remains linearly decodable after filtering, so the filter removes a device-related component that harms transfer rather than all measurement information. The retained bands improve window AUROC on thirteen of fourteen public task–cohort cells (Holm-adjusted p=0.031) and correlation on fourteen of fifteen clinical endpoints in an external cohort of 9,303 participants.

    Submission history

    From: Imran Razzak [view email]
    [v1] Thu, 8 Oct 2026 09:14:26 UTC (2,902 KB)

  • UxSID: Semantic-Aware User Interests Modeling for Ultra-Long Sequence

    UxSID: Semantic-Aware User Interests Modeling for Ultra-Long Sequence

    [Submitted on 9 May 2026 (v1), last revised 9 Oct 2026 (this version, v4)]

    View PDF HTML (experimental)

    Abstract:Modeling ultra-long user sequences involves a difficult trade-off between efficiency and effectiveness. While current paradigms rely on either item-specific search or item-agnostic compression, we propose UxSID, a framework exploring a third path: semantic-group shared interest memory. By utilizing Semantic IDs (SIDs) and a dual-level attention strategy, UxSID captures target-aware preferences without the heavy cost of item-specific models. This end-to-end architecture balances computational parsimony with semantic awareness, achieving state-of-the-art performance and a 0.337% revenue lift in large-scale advertising A/B test.

    Submission history

    From: Hongwei Zhang [view email]
    [v1] Sat, 9 May 2026 16:26:48 UTC (2,829 KB)
    [v2] Wed, 13 May 2026 07:20:14 UTC (2,828 KB)
    [v3] Mon, 18 May 2026 07:00:17 UTC (2,828 KB)
    [v4] Fri, 9 Oct 2026 08:29:36 UTC (2,829 KB)

  • Constrained Triple-Head Price Optimization (C3PO) Network for Discrete-Choice Models

    Constrained Triple-Head Price Optimization (C3PO) Network for Discrete-Choice Models

    [Submitted on 7 May 2026 (v1), last revised 8 Oct 2026 (this version, v2)]

    View PDF HTML (experimental)

    Abstract:We propose a Constrained Triple-Head Price Optimization (C3PO) network for real-time bi-level decision making in discrete-choice environments. In these settings, a seller selects product or service offerings to achieve business objectives, while heterogeneous users make personalized acceptance or rejection decisions based on their preferences, ultimately determining seller outcomes. Existing approaches that fine-tune pre-trained tabular foundation models on synthetic data are limited by their imitation-learning paradigm and inability to enforce common pricing rules, including cross-product and global constraints. C3PO addresses these limitations through a novel transformer-based architecture that jointly integrates price imitation learning, multi-task revenue prediction, and in-context elasticity learning to generate robust pricing recommendations across domains without retraining. A lightweight differentiable constraint layer enables generated prices to satisfy business constraints while optimizing the revenue objective. We demonstrate strong out-of-distribution performance on simulated, synthetic, and real-world datasets. Across simulated benchmarks where the calibrated MNL model is misspecified relative to the ground-truth choice model, C3PO improves expected revenue by 1.4 to 7.9 percentage points over a calibrated MNL baseline. The constraint layer reduces average absolute constraint violations by more than 50%. We further deploy the model in real-world applications, including healthcare tender pricing and airline ancillary pricing, where C3PO improves directional pricing effectiveness, measured by price-increase and price-decrease recall, by 18–34 percentage points depending on the baselines considered, including calibrated MNL, classical optimization, and off-the-shelf tabular foundation-models.

    Submission history

    From: Yingdong Lu [view email]
    [v1] Thu, 7 May 2026 20:55:47 UTC (1,034 KB)
    [v2] Thu, 8 Oct 2026 18:22:06 UTC (1,047 KB)

  • CacheScope: Exact-Compute Allocation for Training-Free Diffusion Acceleration

    CacheScope: Exact-Compute Allocation for Training-Free Diffusion Acceleration

    [Submitted on 8 Oct 2026]

    View PDF HTML (experimental)

    Abstract:Training-free feature reuse accelerates diffusion transformer inference by replacing full denoising computation at selected steps with cached approximations. Existing methods mainly improve the reconstruction of skipped features, but we find that once a stable first-order reconstructor is used, increasing reconstruction complexity brings limited gains, whereas the placement of exact-computation steps has a much larger effect on fidelity. More importantly, absolute shallow-probe variation is jointly affected by input-trajectory displacement and network response strength, and therefore does not directly reflect cache risk. We address this confounding by normalizing probe-output variation with the corresponding input displacement, yielding a directional gain per unit trajectory movement. Based on this signal, we propose CacheScope, which identifies high-risk reuse steps online and selectively falls back to full denoising computation while leaving the underlying cache reconstructor unchanged. CacheScope requires no additional training, gradients, auxiliary networks, or prompt-specific offline schedule optimization. Across FLUX.1-dev, Wan2.1-T2V-1.3B, and Open-Sora 1.2, CacheScope achieves up to 3.57x acceleration and reduces median LPIPS by up to 70.1% over the strongest probe-based baseline.

    Submission history

    From: Qinhan Lyu [view email]
    [v1] Thu, 8 Oct 2026 10:19:34 UTC (2,655 KB)