Investigating LLM Capabilities on Long Context Comprehension for Clinical Question Answering

investigating-llm-capabilities-on-long-context-comprehension-for-clinical-question-answering

Investigating LLM Capabilities on Long Context Comprehension for Clinical Question Answering

[Submitted on 21 Oct 2025 (v1), last revised 12 Nov 2026 (this version, v3)]

View PDF HTML (experimental)

Abstract:Despite the impressive reported performance of Large Language Models (LLMs) on medical Question-Answering (QA), evaluation still overrelies on multiple-choice and extractive settings primarily assessing textbook knowledge, with limited work on open-ended QA over patient records. We present a comprehensive study investigating LLM long-context (LC) comprehension on longitudinal, patient-centric Electronic Health Records (EHRs) across multiple datasets, models and metrics, covering extractive, multiple-choice, and open-ended generative QA under different context formulation and Retrieval Augmented Generation (RAG) settings. We show that medically fine-tuned models offer no measurable benefit over base counterparts, smaller models exhibit stronger memorization, and longitudinal supplementary notes aid generative QA. Reasoning models gain on complex multi-note QA under LLM Correctness but not stricter reference-aligned metrics, while hybrid RAG with reranking outperforms Full-Context (FC) prompting. A question-category analysis and multi-faceted metric evaluation further reveal where RAG and FC excel and struggle, with RAG falling behind in longitudinal reasoning, thus uncovering systematic failures pointing to the need for reasoning-appropriate metrics.

Submission history

From: Feras AlMannaa [view email]
[v1] Tue, 21 Oct 2025 14:50:24 UTC (2,482 KB)
[v2] Wed, 14 Jan 2026 21:40:27 UTC (2,585 KB)
[v3] Thu, 12 Nov 2026 22:58:31 UTC (2,747 KB)