Can gzip be a language model?

The concept of language modeling without neural networks, specifically using an unbounded n-gram model for text generation, has been explored. This approach, which relies on counting rather than weights or training, aligns with the idea that language modeling is compression. The "compression–prediction equivalence" suggests that the score of a candidate can be determined by the length of its gzipped form when combined with its context. Although the implementation uses zlib, which shares the DEFLATE algorithm with gzip, the name GziPT was chosen for its appeal.

This report uniquely details the compression–prediction equivalence as the core principle behind using gzip for language modeling, unlike other discussions that might focus solely on n-gram models.