[Submitted on 20 Nov 2025 (v1), last revised 21 Oct 2026 (this version, v3)]
Abstract:Modern data processing workflows frequently encounter ragged data: multidimensional collections whose inner lengths vary and are revealed only at runtime. Ragged data is prevalent in domains throughout science and computing, yet existing workflow engines treat it as nested containers. This representation leaves shape semantics ambiguous, obscures task dependencies when upstream coordinates are not yet enumerated, and imposes coarse synchronization barriers that stall independent work across tasks. We argue that ragged data is fundamentally a flat function over a coordinate space, and that making this view explicit resolves all three difficulties at once. We realize this insight in Operon, a statically verifiable workflow engine built around a formalism of named dimensions with explicit dependency relations. Users declare pipelines in the Operon DSL by annotating data flow with dimension names, where the static checker verifies that every task does not include faulty iterations regardless of runtime lengths. At runtime, the system models in-progress arrays as partial functions over downward-closed coordinate domains, which provides a natural representation for partially-unknown data states, an exact notion of per-task indegree before all upstream coordinates are enumerated, and a sound basis for fault recovery without programmer intervention.
Submission history
From: Minhyeong Lee [view email]
[v1] Thu, 20 Nov 2025 06:16:31 UTC (132 KB)
[v2] Fri, 16 Oct 2026 02:57:09 UTC (189 KB)
[v3] Wed, 21 Oct 2026 06:32:07 UTC (189 KB)
