[Submitted on 8 Apr 2025 (v1), last revised 15 Oct 2026 (this version, v2)]
Abstract:Despite the success of Large Language Models (LLMs) in long-context tasks, their reliability in recognizing unanswerable questions remains a critical challenge. In information-seeking scenarios, models often suffer from overconfidence and hallucinations when evidence is missing or misleading. In this paper, we propose FACTGUARD, a systematic multi-task workflow for generating evidence-grounded answerable and unanswerable questions. We introduce FACTGUARD-BENCH, a benchmark comprising 25,220 examples with context lengths up to 128K, specifically targeting Lack of Evidence and Misleading Evidence scenarios. Evaluations of eleven representative open-source and proprietary LLMs reveal a significant reliability gap: models struggle to reject unanswerable questions as context scales, with the leading model achieving only 71.62% accuracy. By training with FACTGUARD-BENCH, we improve overall accuracy to 81.17% and significantly enhance models’ reasoned rejection capabilities. Our work provides a scalable path toward building more faithful and trustworthy long-context LLMs. Our source is publicly available at this https URL.
Submission history
From: Qian-Wen Zhang [view email]
[v1] Tue, 8 Apr 2025 01:45:16 UTC (1,225 KB)
[v2] Thu, 15 Oct 2026 08:24:58 UTC (4,070 KB)
