You could have invented transformers from memory

The ability of large language models to recall obscure facts, such as game rules, highlights their vast knowledge. This recall, however, can worsen as the number of memories grows. The interference term in memory retrieval, represented by $P_{jj} - P_{jj}^2$, is approximately $\frac{d_k}{M} - \parens*{\frac{d_k}{M}}^2$. The ratio of interference to signal is approximately $\sqrt{\frac{M}{d_k} - 1}$, where the $-1$ signifies interference canceled by an optimal store.

This analysis quantifies the interference term in memory retrieval as $P_{jj} - P_{jj}^2$, unlike other discussions that only qualitatively describe memory recall.