[Submitted on 13 Aug 2026 (v1), last revised 6 Nov 2026 (this version, v3)]
Abstract:Modern language models are trained on heterogeneous web-scale text corpora. Consequently, studying knowledge and skill acquisition is difficult, as prior exposure to related content is hard to characterize. To address this challenge, we introduce LittleCurriculum, a curated 88B-token pretraining corpus tailored to U.S. elementary school material, explicitly excluding concepts, facts, and vocabulary taught above Grade 5. Training a 5B-parameter LLM from scratch on LittleCurriculum yields LittleLearner, a model with sufficient language competence for open-ended evaluation, yet with clear knowledge and capability boundaries mapped to interpretable curriculum guidelines. We release LittleCurriculum and LittleLearner as a developmentally restricted sandbox to study how models acquire, represent, and use data under a well-defined training scope. We illustrate the sandbox’s utility in a first suite of experiments on injecting new knowledge through post-training and in-context learning. These methods let LittleLearner better utilize existing knowledge, but do not raise out-of-scope capabilities. Our findings underscore the value of this controlled environment for future investigations.
Submission history
From: Jana Zeller [view email]
[v1] Thu, 13 Aug 2026 17:56:12 UTC (2,635 KB)
[v2] Wed, 7 Oct 2026 10:06:13 UTC (2,635 KB)
[v3] Fri, 6 Nov 2026 13:27:58 UTC (2,644 KB)
