Transformers vs RNNs vs SSMs: Where Does Memory Actually Live? [D]

The discussion explores where memory resides in AI architectures like RNNs, Transformers, and SSMs, viewing them through the lens of working memory. It questions whether memory is a compact recurrent state, a growing KV cache, or integrated within the network itself. An example, BDH (Dragon Hatchling), uses linear attention and a low-rank GPU implementation, where recurrent attention state is an N × D matrix. This approach suggests a synaptic interpretation of working memory, aligning it with learned connectivity, though fixed-size states still have finite information capacity.

This report uniquely frames the comparison of RNNs, Transformers, and SSMs by focusing on where memory actually lives, unlike typical architectural horse races.