[Submitted on 3 Nov 2025 (v1), last revised 19 Oct 2026 (this version, v4)]
Abstract:Most engineering problems involve complex multi-step derivations through checkable intermediate quantities. Grading only the final answer misjudges LLMs, crediting correct answers reached through flawed steps and penalizing wrong answers built on mostly sound ones. Yet, existing benchmarks such as SuperGPQA and EEE-Bench score only final answers, and step-level ones such as TPS-CalcBench rely on LLM judges or human graders, or cover a single domain. We therefore introduce EngTrace, a symbolic benchmark for verifiable process evaluation of engineering reasoning. Its 150 expert-certified templates, spanning five branches, 15 domains, and 42 areas, generate each problem with a gold reasoning trace, so that every intermediate quantity is known exactly. We design and validate an evaluator that checks final answers, coverage of the gold trace’s intermediate quantities (milestones), and displayed arithmetic, mostly without an LLM judge. Across 11 LLMs, final answers barely separate the strongest models, while their derivations differ. Wrong answers still state about half of the gold milestones, and domain experts trace most wrong answers to calculation slips, with conceptual errors rarer on easy problems than on harder ones. Overall, as LLMs near the final-answer ceiling, checking derivations, as EngTrace does, reveals differences that final answers miss.
Submission history
From: Ayesha Gull [view email]
[v1] Mon, 3 Nov 2025 15:05:44 UTC (5,583 KB)
[v2] Wed, 7 Jan 2026 15:44:03 UTC (2,221 KB)
[v3] Tue, 16 Jun 2026 12:04:30 UTC (2,319 KB)
[v4] Mon, 19 Oct 2026 15:46:42 UTC (4,581 KB)
