[Submitted on 3 May 2024 (v1), last revised 9 Oct 2026 (this version, v3)]
Abstract:This paper introduces OARelatedWork: a dataset for related work generation from open-access sources. It is the first large-scale multi-document summarization dataset for related work generation, containing whole related work sections and full texts of cited papers. Its validation and test splits are constructed so that every source paper is available in full text, enabling controlled evaluation of full-text related work generation. The dataset includes 94 450 papers from multiple domains. With OARelatedWork, we aim to shift the field from generating parts of related work sections from abstracts only to generating entire related work sections from all available content. We (i) benchmark a wide spectrum of models, highlighting that synthesizing massive full-text contexts remains challenge for tested cost-effective Large Language Models (LLMs): under our statement-level judge, GPT-4o-mini’s evidence-grounded True rate drops from 92.9% with abstracts to 83.8% with full texts. We (ii) empirically analyze human writing behavior through a human evaluation over 40 papers and 408 factual statements, revealing that authors frequently introduce abstractive claims ungrounded in localized source texts; consequently, tested LLMs actually surpass human baselines in strict, evidence-grounded factuality. Finally, we (iii) conduct a fine-grained meta-evaluation, revealing that standard reference-based metrics are inadequate for evaluating such long-form structured outputs, and introduce a robust statement-level evaluation framework to address this gap.
Submission history
From: Martin Docekal [view email]
[v1] Fri, 3 May 2024 08:49:22 UTC (20,532 KB)
[v2] Mon, 1 Jun 2026 07:12:15 UTC (10,502 KB)
[v3] Fri, 9 Oct 2026 07:25:55 UTC (10,511 KB)