artificialintelbulletin.com

Tag: cs.CY

  • ABLEIST: Intersectional Disability Bias in LLM-Generated Hiring Scenarios

    ABLEIST: Intersectional Disability Bias in LLM-Generated Hiring Scenarios

    [Submitted on 13 Oct 2025 (v1), last revised 8 Oct 2026 (this version, v2)]

    View PDF HTML (experimental)

    Abstract:Large language models (LLMs) are increasingly under scrutiny for perpetuating identity-based discrimination in high-stakes domains such as hiring, particularly against people with disabilities (PwD). However, existing research remains largely Western-centric, overlooking how intersecting forms of marginalization–such as gender and caste–shape experiences of PwD in the Global South. We conduct a comprehensive audit of six LLMs across 2,820 hiring scenarios spanning diverse disability, gender, nationality, and caste profiles. To capture subtle intersectional harms and biases, we introduce ABLEIST (Ableism, Inspiration, Superhumanization, and Tokenism), a set of five ableism-specific and three intersectional harm metrics grounded in disability studies literature. Our results reveal significant increases in ABLEIST harms towards disabled candidates–harms that many state-of-the-art models failed to detect. These harms were further amplified by sharp increases in intersectional harms (e.g., Tokenism) for gender and caste-marginalized disabled candidates, highlighting critical blind spots in current safety tools and the need for intersectional safety evaluations of frontier models in high-stakes domains like hiring.

    Submission history

    From: Hayoung Jung [view email]
    [v1] Mon, 13 Oct 2025 04:18:23 UTC (1,493 KB)
    [v2] Thu, 8 Oct 2026 22:37:37 UTC (1,389 KB)

  • The One-Word Census: Answer-Choice Conformity Across 105 Language Models

    The One-Word Census: Answer-Choice Conformity Across 105 Language Models

    [Submitted on 14 Jul 2026 (v1), last revised 9 Oct 2026 (this version, v4)]

    View PDF HTML (experimental)

    Abstract:When a language model must choose one answer from a large space of equally valid options, which answer does it choose, and how often is it the answer every other model chooses? Asked to “pick a word,” 105 language models from more than twenty labs chose serendipity 46% of the time. We measure this convergence, and each model’s share in it, with 96 single-turn prompts that each name a category with many valid one-word answers (“Name a tree.”), asked eight times per model and scored by exact match, with no embeddings and no judge. A model’s answer-choice surprisal is the average -log2 probability of its answers under the pooled answers of all other models. In 28 of 96 categories a single answer takes at least 80% of all answers. The concentration does not depend on small or persona-tuned models: the 87 major-lab models are at least as concentrated as the full field. Lightly post-trained and persona-tuned models are the most divergent; heavily post-trained assistants from the major labs are the most conformist. Models that avoid the modal answer mostly land on the same runner-up. Within the major providers’ lineages, release order shows no panel-wide trend once model tier is controlled; GPT, Gemini, Grok and Qwen become more conformist across releases, and Claude’s generation-5 releases reverse. On open checkpoints of three post-training pipelines, supervised fine-tuning is the largest step toward the field’s answers. Against human category-production norms, the field is more concentrated than people in 18 of 20 shared categories. All prompts, transcripts, and code are public.

    Submission history

    From: Tapan Parikh [view email]
    [v1] Tue, 14 Jul 2026 14:12:05 UTC (293 KB)
    [v2] Sat, 25 Jul 2026 18:18:32 UTC (297 KB)
    [v3] Wed, 7 Oct 2026 20:52:36 UTC (123 KB)
    [v4] Fri, 9 Oct 2026 14:01:46 UTC (123 KB)

  • Activating the Annotator: Operationalising Lived Experience in Data Annotation

    Activating the Annotator: Operationalising Lived Experience in Data Annotation

    [Submitted on 8 Oct 2026]

    View PDF HTML (experimental)

    Abstract:Perspectivist NLP has established that annotator positionality shapes labels for subjective tasks, but the field still lacks methods for operationalising it. Positionality is typically operationalised as a demographic form completed at recruitment: a static attribute recorded once and treated as a covariate, on the assumption that the knowledge it indexes is active when the annotator reads an item. This paper treats that assumption as a design problem and asks whether lived experience can instead be primed by the task itself. We introduce AXLE, a pre-annotation stage in which a scenario vignette anchors a short, axis-guided conversation that is synthesised into a first-person micro-narrative and returned to the annotator before labelling begins. In a 2×2 between-subjects study (104 annotators, 30 items, 1,040 annotations) on ambiguous hate speech spanning immigration, religion and gender, elicitation was associated with higher identity resonance and higher sensitivity to harm. It also increased label entropy and lowered inter-annotator agreement: the stage appears to surface plurality rather than manufacture consensus, which distinguishes it from annotator training, whose success criterion is convergence.

    Submission history

    From: Sheza Munir [view email]
    [v1] Thu, 8 Oct 2026 20:31:56 UTC (566 KB)

  • DistressBench: Evaluating Multi-Dimensional Crisis Support in Large Language Models

    DistressBench: Evaluating Multi-Dimensional Crisis Support in Large Language Models

    [Submitted on 8 Oct 2026]

    View PDF HTML (experimental)

    Abstract:Language models have been adopted faster than evaluation practices that reliably assess their risks, and a growing share of use is affective rather than informational. Within that traffic users disclose suicide and self-harm (SSH) crises. Existing safety benchmarks score models primarily on refusal, but refusal is not the desired behavior here; the conversation requires a substantive supportive response. We introduce DistressBench, 718 English conversations with clinician-authored user turns spanning 23 SSH subcategories, each paired with a task-specific rubric of weighted atomic criteria across seven dimensions of crisis support and a clinician reference response. A model’s score is the weighted average over the criteria its reply satisfies, reading directly as the fraction of required care delivered. The best of 25 frontier models satisfies 88.4% of weighted criteria and the median model 71.4%, against 98.7% for clinician references scored on the same criteria. Pooled across all 25 models, in 35.3% of conversations where a model satisfied every criterion for recognizing the crisis, it failed to route the user toward human help. The shortfall concentrates in the dimensions requiring the model to act rather than withhold. Contrasting each dimension’s median model against its best separates capability limits from choices: de-escalation reaches 74.6% in the strongest model against a median of 46.0%, and safety disclaimers 84.6% against 8.3%. Multi-turn conversations score below single-turn ones (68.5% against 74.2%). We release the corpus, rubrics, judge calibration labels, and a companion annex of 20 long-horizon red-team transcripts. These results argue for evaluating crisis safety on the care a model delivers rather than on what it declines to say, and for treating de-escalation and resource routing as deployment requirements.

    Submission history

    From: Drew Rein [view email]
    [v1] Thu, 8 Oct 2026 22:09:12 UTC (328 KB)