[Submitted on 10 Nov 2026 (v1), last revised 11 Nov 2026 (this version, v2)]
Abstract:Language-model systems in rule-governed domains often pair the model with a deterministic fact layer, but what that layer should compute is rarely measured. We measure it in the Chinese sexagenary system underlying BaZi chart analysis, where every fact has one table-checkable value: 682 items over twelve fact categories and 160 transit–natal collision items, each posed on a ladder of rungs that differ only in how much of the computation is done for the model. Answering directly, none of eleven open-weight models gets more than 4% of sexagenary days or 1% of collisions right, and supplying the attributes of every pillar does not change the collisions. A complete list of computed relations is not enough either: it leaves Qwen3-32B at 1%, with relations listed for other pairs misattributed to the queried pair, whereas a query-filtered layer from follow-up runs, stating only the queried relations pillar by pillar, gives 92%. On chess positions, two predictions locked in advance hold: attributes do not help, and unqueried relations are again misattributed. Three proprietary reasoning models compute 68–95% of collisions unaided, while reasoning in smaller open models overrides supplied facts. A fact layer should compute the interactions and return the ones the question asks about.
Submission history
From: Huixue Han [view email]
[v1] Tue, 10 Nov 2026 05:15:58 UTC (352 KB)
[v2] Wed, 11 Nov 2026 08:29:19 UTC (354 KB)
