[Submitted on 19 May 2026 (v1), last revised 14 Oct 2026 (this version, v2)]
Abstract:Automatic evaluation of long-form literary writing remains challenging, particularly for creativity-related dimensions such as originality and flexibility. We address the lack of large-scale supervision for structured Torrance Test of Creative Writing (TTCW)-based evaluation by constructing a dataset of 263{,}911 long-form stories, each paired with scalar scores and meta-synthesised review comments across 14 TTCW-based dimensions. Using this dataset, we study whether reviewer-style reasoning traces improve structured literary review generation across Qwen3-4B, Qwen3-8B, and Nemotron-8B. Across all three models, no-trace supervision yields higher score imitation fidelity and near-complete report generation, whereas trace supervision reduces completion to $85.9$–$90.9%$. Failure analysis shows that $88.6%$ of failed reasoning generations contain semantic repetition loops inside the trace. These loops concentrate in particular story–criterion interactions and progressively harden after onset, while comparable repetition is rare in the supervision data. Our results suggest that, for long-form tasks requiring strictly structured outputs, explicit reasoning traces can introduce generation instability that outweighs their overall benefit.
Submission history
From: Jinlong Liu [view email]
[v1] Tue, 19 May 2026 18:16:58 UTC (222 KB)
[v2] Wed, 14 Oct 2026 11:05:55 UTC (321 KB)
