[Submitted on 30 May 2025 (v1), last revised 21 Oct 2026 (this version, v3)]
Abstract:Large Language Models (LLMs) have shown potential in simulating human behaviors and performing theory-of-mind (ToM) reasoning, crucial for complex social interactions. We investigate ToM reasoning’s role in aligning agentic behaviors with human norms in negotiation tasks, using the ultimatum game as our referenced task. We initialized LLM agents with different prosocial beliefs (Greedy, Fair, Selfless) and reasoning methods (chain of thought and ToM reasoning of varying levels), examining their decision-making processes and outcomes across multiple LLMs, including reasoning models like o3-mini and DeepSeek-R1 Distilled Qwen 32B. We perform 2,700 simulations to show that ToM reasoning enhances behavioral alignment with human norms, decision-making consistency, and negotiation outcomes. Consistent with prior findings, reasoning LLMs exhibit limited capability compared to ToM-enhanced LLMs, with different game roles benefiting from different ToM orders. Agents almost always chose strategies consistent with their assigned beliefs, and doing so aligned better with human norms in nearly every role, yet Greedy and Selfless agents still fell short of their norms in outcomes. Human verification further revealed that Llama 3.3 70B produces reasoning most consistent with its actions and beliefs. Our findings advance understanding of ToM’s role in human-AI interaction and cooperative decision-making. The code used for our experiments can be found at this https URL.
Submission history
From: Palakorn Achananuparp [view email]
[v1] Fri, 30 May 2025 06:23:52 UTC (3,380 KB)
[v2] Mon, 24 Aug 2026 03:10:58 UTC (156 KB)
[v3] Wed, 21 Oct 2026 12:23:50 UTC (155 KB)
