[Submitted on 12 Apr 2026 (v1), last revised 26 Oct 2026 (this version, v4)]
Abstract:Game engines hold what video models struggle to learn: a complete, explicit world state behind every frame. We turn one into a data instrument. GEST-Engine, our production-grade open-source system, deterministically executes Graphs of Events in Space and Time (GESTs), whether procedurally generated or derived from text, into videos of synchronized multi-actor scenarios, recording ground truth as it renders: 3D entity and camera state, pairwise spatial relations, event-to-frame mappings, instance segmentation, and long descriptions, at zero marginal annotation cost. With it we release GTASA, a 938-video sample of what the system can generate at arbitrary scale, carrying, to our knowledge, the densest spatial-relation coverage of any video dataset: a complete entity-pair relation graph at every frame, ~84x denser than the state of the art, frame-for-frame. We validate GTASA both qualitatively, through human evaluation of physical validity and semantic alignment where frontier neural generators, given the same prompts, largely fail, and quantitatively, with GTASA pretraining improving VideoLLaMA3 video captioning. Probing six frozen video encoders across 11 spatio-temporal tasks enabled by GTASA’s exact 3D ground truth, a previously untestable inter-entity relational probe of frozen video features, reveals that who-is-near-whom barely rises above chance for all of them. We release the engine, the corpus, and the benchmark, making this gap a measurable, trainable target.
Submission history
From: Nicolae Cudlenco [view email]
[v1] Sun, 12 Apr 2026 00:01:51 UTC (201 KB)
[v2] Mon, 13 Jul 2026 16:39:12 UTC (9,135 KB)
[v3] Tue, 14 Jul 2026 05:51:48 UTC (9,135 KB)
[v4] Mon, 26 Oct 2026 22:03:11 UTC (6,885 KB)
