[Submitted on 7 May 2026 (v1), last revised 6 Nov 2026 (this version, v2)]
Abstract:Large language models have demonstrated impressive retrieval-augmented capabilities. However, a crucial area remains underexplored: their ability to appropriately adapt responses to the certainty of the retrieved information. It is a limitation with real consequences in high-stakes domains like medicine and finance. We evaluate eight LLMs on their context-certainty obedience, measuring how well they adjust responses to match expressed context certainty. Our analysis reveals systematic limitations: LLMs fail to reproduce their prior outputs after observing an uncertain context, respond to expressed certainties in a distorted way, and overrely on complex contexts. To address these, we propose an interaction strategy combining prior reminders (feeding the model’s own context-free answer back into the prompt), context-certainty recalibration, and context simplification. This approach reduces obedience errors by 25% on average, without modifying model weights, demonstrating the efficacy of interaction design in enhancing LLM reliability. Our contributions include a principled evaluation metric, empirical insights into LLMs’ uncertainty handling, and a portable strategy to improve context-certainty obedience across diverse LLMs.
Submission history
From: Behzad Shayegh [view email]
[v1] Thu, 7 May 2026 20:29:22 UTC (359 KB)
[v2] Fri, 6 Nov 2026 17:37:04 UTC (472 KB)
