Why don't machine learning research agents overfit?

Machine learning aims for generalization, not memorization, to perform well on new data rather than just training examples; failure to do so is called overfitting. LLM-based research agents, like human communities, also engage in benchmark hill-climbing without overfitting. A recent paper, "What fits (into few tokens) doesn't overfit: Compression and generalization in ML research agents," explains this by demonstrating that these agents can achieve strong performance with remarkably small compressions, such as 32-token prompts across eight datasets, or even 16 tokens for one language-modeling strategy, without loss in performance.

This paper offers a concrete explanation for why LLM-based research agents do not overfit, unlike previous theories that only observed the phenomenon.