Analysis: from 2019 to 2025, gains in pretraining compute efficiency came mostly from data improvements rather than model improvements (Dwarkesh Podcast)

From 2019 to 2025, improvements in pretraining compute efficiency primarily stemmed from advancements in data rather than model enhancements, according to an analysis discussed on the Dwarkesh Podcast. This breakdown examines six years of pretraining progress, highlighting how data improvements have been the main driver behind the rapid advancements observed in AI during this period, rather than breakthroughs in model architecture or algorithms themselves.