[Submitted on 22 May 2026 (v1), last revised 29 Oct 2026 (this version, v2)]
Abstract:Fixed game suites for evaluating strategic reasoning in large language models (LLMs) saturate and are hard to keep free of contamination. We introduce GENSTRAT, a system for evaluating the strategic capabilities of LLMs driven by procedural generation of two-player zero-sum imperfect-information betting games, allowing an evaluator to draw novel, fresh games on demand. Each game is characterized on a scale of five complexity axes (state space, temporal contingency, opponent dependence, private information, precision) and measured by Monte Carlo simulation. Using this generator, we construct a 50-game benchmark and hold a tournament of twelve frontier and open-weight models, gathering data from over 40,000 LLM vs. LLM matches. We further construct a composite complexity axis, and find that, on the eight hardest games, the gap between the best and worst model is 1.7x its value across the whole set of games. Ablations show that a higher thinking budget, disclosing the opponent’s strategy, and higher reasoning matched to a game’s specific demands (for private information and planning depth) were associated with higher margins for the LLM player.
Submission history
From: Vartan Shadarevian [view email]
[v1] Fri, 22 May 2026 05:13:45 UTC (1,940 KB)
[v2] Thu, 29 Oct 2026 23:30:39 UTC (1,122 KB)
