I gave a 21M model a 6.4B-parameter lookup table. It matches a 114M dense model and runs with the table on an SSD (RX 9070)

A hobby research project has made public a 21M model that, when paired with a 6.4B-parameter lookup table, performs comparably to a 114M dense model. This setup, which uses 33M parameters per token from the 16.8M-row table, runs efficiently with the table stored on an SSD (RX 9070). The developer is seeking feedback and is interested in scaling the project to 1B if larger GPUs are available.

This project demonstrates a novel approach where a small 21M model, unlike larger dense models, achieves comparable performance by offloading a 6.4B-parameter lookup table to an SSD.