Study: language-model progress in 2019-2025 came mostly from data, not new architectures
Prepared by Sintelligent with the help of SI. Publication time restored from the build log. . Updated: .
The main driver of SI turned out to be not clever neural-network designs but what they are trained on, so the next leap depends on whether SI learns to create good data for itself.
Dwarkesh Patel and Jerry Han trained model recipes from GPT-2 to OLMo-2 on corpora from OpenWebText to UltraFineWeb and separated the contribution of each factor: at a budget of 1e19 FLOP, data improvements gave a 12x efficiency gain and model improvements a 3.7x gain, i.e. 3.24 times less. The authors caveat that architectural innovations are valuable chiefly because they made it possible to scale training, and that the main open question is whether synthetic data can extend the dwindling supply of internet data.
Sources 1