artificialintelbulletin.com

Author: admin

  • Heterogeneity-Aware Client Sampling for Optimal and Efficient Federated Learning

    Heterogeneity-Aware Client Sampling for Optimal and Efficient Federated Learning

    [Submitted on 16 May 2025 (v1), last revised 9 Oct 2026 (this version, v4)]

    View PDF HTML (experimental)

    Abstract:Federated learning (FL) commonly involves clients with diverse communication and computational capabilities. Such heterogeneity can significantly distort the optimization dynamics and lead to objective inconsistency, where the global model converges to an incorrect stationary point potentially far from the pursued optimum. Despite its critical impact, the joint effect of communication and computation heterogeneity has remained largely unexplored, due to the intrinsic complexity of their interaction. In this paper, we reveal the fundamentally distinct mechanisms through which heterogeneous communication and computation drive inconsistency in FL. To the best of our knowledge, this is the first unified theoretical analysis of general heterogeneous FL, offering a principled understanding of how these two forms of heterogeneity jointly distort the optimization trajectory under arbitrary choices of local solvers. Motivated by these insights, we propose Federated Heterogeneity-Aware Client Sampling, FedACS, a universal method to eliminate all types of objective inconsistency. We theoretically prove that FedACS converges to the correct optimum at a rate of $O(1/sqrt{R})$, even in dynamic heterogeneous environments. Extensive experiments across multiple datasets show that FedACS outperforms state-of-the-art and category-specific baselines by 4.3%-36%, while reducing communication costs by 22%-89% and computation loads by 14%-105%, respectively.

    Submission history

    From: Shudi Weng [view email]
    [v1] Fri, 16 May 2025 14:31:36 UTC (1,124 KB)
    [v2] Mon, 16 Feb 2026 06:46:40 UTC (585 KB)
    [v3] Sun, 22 Feb 2026 05:58:59 UTC (580 KB)
    [v4] Fri, 9 Oct 2026 01:27:04 UTC (1,377 KB)

  • Enriched Holomorphic Neural Networks for 2D and 3D Laplace Problems

    Enriched Holomorphic Neural Networks for 2D and 3D Laplace Problems

    [Submitted on 8 Oct 2026]

    View PDF HTML (experimental)

    Abstract:Boundary-value problems for the Laplace equation develop singular behaviour near corners, edges and junctions that smooth approximators resolve poorly. We combine holomorphic neural networks with singular functions derived from the local geometry and boundary conditions. The representation is harmonic by construction, so no interior residual is needed: the network fits the smooth remainder from boundary data alone, and the enrichment terms capture the singular behaviour. Singular exponents are trainable, so the construction also applies when the local expansion is not known in advance. We extend the approach to three dimensions using independent networks evaluated along complex null directions. The enriched model is the most accurate tested method on every problem. Its margin is several orders of magnitude where the enrichment captures the singularity exactly, and a factor of about 1.7-2.1 over the next-best method on a CdZnTe radiation detector with many singular sites, in both two and three dimensions.

    Submission history

    From: Alejandro Valverde Mahou [view email]
    [v1] Thu, 8 Oct 2026 14:48:38 UTC (1,890 KB)

  • Quadratic Characterizations for Reachability Analysis of Neural Networks

    Quadratic Characterizations for Reachability Analysis of Neural Networks

    [Submitted on 19 May 2026 (v1), last revised 9 Oct 2026 (this version, v2)]

    View PDF HTML (experimental)

    Abstract:Quadratic constraints (QCs) are widely used to characterize nonlinearities and uncertainties, but generic analytical characterizations can be conservative on bounded domains. This paper develops a framework for constructing verified quadratic characterizations of scalar relations in the two-dimensional real plane. Candidate quadratic inequalities are locally generated by solving convex quadratic programs using samples from the relation and exterior sample points. They are then verified globally using sum-of-squares certificates over an exact semialgebraic description or, in the case of nonpolynomial relations, over relaxed polynomial descriptions. The resulting verified constraints define a sound overapproximation of the scalar relations over the considered domains. These constraints are directly compatible with existing analysis frameworks based on QCs and pointwise integral quadratic constraints (IQCs) for static nonlinearities and uncertainties, and they can also be embedded in QC-based semidefinite programs for reachability and safety analysis of feedforward neural networks. For smooth activations such as $tanh$, the method yields domain-dependent quadratic characterizations that constitute an alternative to generic sector- or slope-based descriptions. For ReLU networks, we give methods to reduce conservatism in QC-based reachability analysis of feedforward networks by exploiting dependencies between neurons and tighter local bounds. Numerical examples demonstrate improved reachability results for smooth activations, reduced conservatism for ReLU networks, and applicability beyond neural networks through an example involving saturation.

    Submission history

    From: Elias Khalife [view email]
    [v1] Tue, 19 May 2026 20:45:31 UTC (735 KB)
    [v2] Fri, 9 Oct 2026 04:57:49 UTC (738 KB)

  • Transformer-Based Masked Diffusion Models Provably Learn Distributions with Latent Structure

    Transformer-Based Masked Diffusion Models Provably Learn Distributions with Latent Structure

    [Submitted on 8 Oct 2026]

    View PDF HTML (experimental)

    Abstract:Masked diffusion models (MDMs) generate discrete data by iteratively unmasking multiple tokens in parallel using a learned mask predictor. Their output carries two errors: parallelization error due to parallel sampling, and estimation error from the mask predictor learned with finite samples. Existing theory treats these errors separately, and the available statistical bounds typically scale with the number $V^L$ of length-$L$ sequences over a vocabulary of size $V$, suffering from the curse of dimensionality. In this work, we establish end-to-end statistical guarantees showing that transformer-based MDMs learn structured distributions with a sample complexity governed by their latent structure. Specifically, we model this structure through mixtures of hidden Markov trees, which capture latent dependencies within components and heterogeneity across components. When the data distribution is well approximated by such a mixture, a transformer-based MDM with parallel unmasking schedules achieves $varepsilon$-accurate sampling in KL divergence with $widetilde{O}(LK_{mathrm{par}}/varepsilon)$ training samples, where $K_{mathrm{par}}$ is polynomial in $L$, $V$, the number of hidden states, and the number of mixture components. The transformer network and sampling procedure require no knowledge of the tree topologies or component parameters, thereby adapting to the unknown latent structure automatically. Overall, our results provide theoretical justification for the empirical effectiveness of MDMs on real-world data.

    Submission history

    From: Jingda Wu [view email]
    [v1] Thu, 8 Oct 2026 17:54:49 UTC (90 KB)

  • Measuring and Mitigating Solution Mode Collapse in RLVR

    Measuring and Mitigating Solution Mode Collapse in RLVR

    [Submitted on 8 Oct 2026 (v1), last revised 9 Oct 2026 (this version, v2)]

    View PDF HTML (experimental)

    Abstract:A language model (LM) can usually answer the same question in more than one way, but reinforcement learning with verifiable rewards (RLVR) is indifferent to which correct answer a model produces. A solution will earn the same reward whether it is the thousandth copy of a familiar answer or one the model has never produced before. Yet, there is potential value in having the model retain multiple correct solutions as it is trained. For instance, multiple modes may give users a choice and provide problem-solving strategies that improve overall model performance. Here, we introduce ModeBench, a benchmark of multi-solution tasks in which the verifier returns both correctness and mode discovered. We then use ModeBench to measure how solution diversity changes under RLVR post-training. We find that RLVR post-training concentrates probability onto fewer correct modes even as accuracy holds or improves, and moreover, that frontier models are already highly concentrated. We then introduce our solution, Re:Max, which stores one verified example per discovered mode in a replay buffer and trains on those stored modes uniformly. A solution found once is, therefore, practiced as often as one found repeatedly. Across three model scales, two RL objectives, and harder task constructions, replay improves both how often a policy succeeds and how many different ways it can succeed.

    Submission history

    From: Liv D’Aliberti [view email]
    [v1] Thu, 8 Oct 2026 01:25:16 UTC (744 KB)
    [v2] Fri, 9 Oct 2026 02:22:30 UTC (744 KB)

  • One Model to Loop Them All: Elastic Latent Recurrence for Language Models

    One Model to Loop Them All: Elastic Latent Recurrence for Language Models

    [Submitted on 8 Oct 2026]

    View PDF HTML (experimental)

    Abstract:A latent recurrent language model applies a shared block of layers repeatedly, so one checkpoint can trade inference compute for accuracy by changing the loop count. At one loop, however, such models typically fall well short of a non-recurrent model of the same size, a one-loop tax that makes a separate non-recurrent model the better choice whenever latency matters. We present Elastic Latent Recurrence (ELR), a recipe for training looped language models without this tax. At one loop, an ELR model costs the same to serve as a non-recurrent model of the same size and, trained on the same data, reaches similar accuracy. Each additional loop then buys more accuracy for more inference compute. Three choices make this work: a gated state update with a learned step size, loop counts mixed within each optimizer update, and conditioning each pass on its loop index and budget. The first two preserve one-loop accuracy, and the third lowers loss at every additional loop. Mixed depths can be introduced late in pretraining to limit the added training cost, and with zero-initialized conditioning added at the switch, this late schedule reaches lower loss than mixed-depth training from scratch. In MoE models, reusing first-loop expert routes on later loops cuts mixed-depth update time by 11.7% with no measurable change in accuracy. At inference, keeping only the last loop’s attention KV cache holds attention KV memory at its one-loop level while retaining most of the gain from additional loops. On 4B dense and 30B-A3B MoE hybrid models, ELR at one loop is 0.52 points below and 1.04 points above its matched baseline on a ten-task average, and four loops add 7.25 and 4.75 points. Carried through post-training, the recipe yields Koi-4B, a single checkpoint that scores an estimated 18 on the Artificial Analysis Intelligence Index at one loop, matching Qwen3.5-4B, and 29 at four loops.

    Submission history

    From: Shantanu Acharya [view email]
    [v1] Thu, 8 Oct 2026 18:05:42 UTC (296 KB)

  • Contextual Correction of Parallel Draftsfor Speculative Decoding

    Contextual Correction of Parallel Draftsfor Speculative Decoding

    [Submitted on 8 Oct 2026]

    View PDF HTML (experimental)

    Abstract:Speculative decoding accelerates large language model inference by using a lightweight drafter to propose multiple tokens that are subsequently verified in parallel by the target model. Recent diffusion-based drafters make this paradigm particularly attractive: deep attention-based neural backbones provide strong predictions, while drafting multiple future positions in parallel avoids the sequential cost of autoregressive generation. This parallelism, however, comes at a cost. Tokens within a draft block are predicted without access to the realized identities of preceding draft tokens, removing intra-block dependencies that would naturally be available under autoregressive generation. We introduce DCOCO (DFlash Contextual Corrector), a lightweight mechanism that restores these dependencies while preserving parallel drafting. Rather than committing to each position’s top prediction, DCOCO retains multiple candidate tokens and evaluates their contextual compatibility across adjacent draft positions in parallel, enabling contextually informed token selection at low additional cost. Across four verifier backbones and ten benchmarks, DCOCO achieves the highest average throughput in every evaluated configuration, spanning greedy and stochastic decoding with and without extended reasoning, with speedups of up to 6.6x over vanilla decoding. Its advantage persists under concurrent serving: DCOCO outperforms the strongest competing method by 6-23% across 4-32 concurrent requests, while other sequential correctors fall below uncorrected DFlash at these loads. These results demonstrate that recovering intra-block dependencies through parallel contextual scoring delivers consistent acceleration across decoding regimes and serving loads.

    Submission history

    From: Roy Eisenstadt [view email]
    [v1] Thu, 8 Oct 2026 18:01:03 UTC (236 KB)

  • How AI Writing Assistance Distorts Perceptions of AI Users and Their Opinions

    How AI Writing Assistance Distorts Perceptions of AI Users and Their Opinions

    [Submitted on 24 Apr 2026 (v1), last revised 8 Oct 2026 (this version, v3)]

    View PDF HTML (experimental)

    Abstract:Hundreds of millions of people use AI for writing assistance. We asked how AI distorts perceptions of AI users and their opinions. Writers (N=2,939) wrote political opinion paragraphs with and without AI assistance. Separate readers (N=11,091) blindly evaluated these paragraphs and made inferences about the writers. With AI, writers seemed more opinionated, competent, positive, and demographically privileged. Writers objected to many of these distortions yet preferred AI-assisted text even when made aware of them. Reward models trained on our data mitigated objectionable distortions but reduced user acceptance. Additional readers (N=8,798) trusted AI-assisted writers more and were more persuaded by AI writing when AI was more distortive. Overall, distortions from AI writing assistance are pervasive, persistent under human oversight, and likely to have consequential effects on human behaviours and attitudes.

    Submission history

    From: Paul Röttger [view email]
    [v1] Fri, 24 Apr 2026 12:31:11 UTC (1,354 KB)
    [v2] Mon, 29 Jun 2026 11:51:27 UTC (1,436 KB)
    [v3] Thu, 8 Oct 2026 19:19:51 UTC (3,683 KB)

  • Smoothing the Top-k Exposure Boundary for Sparse Mixture-of-Experts

    Smoothing the Top-k Exposure Boundary for Sparse Mixture-of-Experts

    [Submitted on 8 Oct 2026 (v1), last revised 9 Oct 2026 (this version, v2)]

    View PDF HTML (experimental)

    Abstract:Sparse Mixture-of-Experts models scale parameter capacity efficiently while maintaining a fixed compute budget per token. However, traditional training paradigms enforce a static choice of top-$k$ experts, which converts a continuous routing distribution into a rigid step function. This constraint introduces a brittle boundary where highly competitive experts are arbitrarily separated into full-supervision and zero-feedback zones based on minor score fluctuations. To address this issue, we propose Elastic Expert Routing, which stochastically samples the active expert budget from a localized discrete distribution centered at $k$. Over multiple training iterations, this mechanism softens the sharp threshold into a gradual probability distribution. Because the sampling neighborhood remains symmetric, this approach matches the expected computational cost of deterministic training, while preserving the inference budget. During supervised fine-tuning, elastic routing improves downstream macro-averages on OLMoE-1B-7B and Qwen3-30B-A3B by $+0.84$ and $+2.02$ points, respectively. In addition, in from-scratch pretraining, it reduces validation LM loss by up to $0.04$ and outperforms the static top-$k$ baseline by $1.6$ points on average across downstream tasks. Comparisons on Qwen3 fine-tuning and from-scratch pretraining further show higher average scores than Noisy top-$k$, expert FFN dropout, and DefaultMoE.

    Submission history

    From: Yunkai Chai [view email]
    [v1] Thu, 8 Oct 2026 09:31:19 UTC (157 KB)
    [v2] Fri, 9 Oct 2026 09:32:21 UTC (159 KB)

  • Unsupervised Multimodal Reward Modeling

    Unsupervised Multimodal Reward Modeling

    [Submitted on 8 Oct 2026]

    View PDF HTML (experimental)

    Abstract:Vision-language reward models are trained on annotated preference pairs, which are expensive to collect, can disagree, and are ranked by benchmark accuracy. We build one with no human preference label: a linear head on an intermediate layer of a frozen Qwen3-VL backbone, trained to score a web caption above a perturbed version of the same caption; labeled data enters only to select the layer. The layer is the decisive choice, worth 5.8 points over the last layer, and one recipe transferred without re-tuning beats or at least matches GPT-4o on three benchmarks at 32B, 6.6 points ahead on VL-RewardBench, from a few thousand parameters at about a thousandth of fine-tuning compute. At an equal pair budget, perturbations teach a frozen head 3 to 5 points more than curated RLAIF-V pairs. Because the backbone is frozen, the objective is convex, and the perturbation sampler specifies the reward, which the benchmark does not identify: two heads trained on disjoint halves of the same perturbations tie on the benchmark and differ by 6.4 points in hallucination judging. Retraining the head on the policy’s own verifier-labeled candidates cuts caption hallucination by 5.1 CHAIR points and lifts image-generation pass rate by 5.4, while scoring lower on every reward benchmark.
    Together, an intermediate-layer probe, free same-image perturbations, and a verifier-driven bootstrap turn give a recipe for multimodal reward modeling that needs no annotator, costs a fraction of fine-tuning, and is specified by the comparisons it is trained on rather than by a benchmark rank.

    Submission history

    From: Yueying Li [view email]
    [v1] Thu, 8 Oct 2026 20:38:25 UTC (957 KB)