Experimenting with an event-driven 1B architecture on consumer hardware

I’m a solo developer experimenting with alternative neural network architectures, and I wanted to share a project I’ve been working on recently.

For some time I’ve been exploring SSN-inspired models and event-driven computation. My latest experiment is an architecture I call RHEA.

RHEA is built around event-driven responses rather than trying to reproduce a conventional Transformer stack. The architecture does not rely on the standard attention/MLP block structure used by most Transformer language models, and I’m experimenting with a different internal processing approach.

One of my main goals is to make training relatively large experimental models possible on consumer hardware.

I’ve also been working on memory and compute optimizations that currently allow me to experiment with a roughly 1B-parameter RHEA model on a laptop with an RTX 5070 Laptop GPU with 8 GB of VRAM.

That hardware limitation is actually part of the experiment: I’m interested in seeing how far this type of architecture can be pushed without access to large GPU clusters.

It’s still very much a research project, so I’m not claiming that it replaces Transformers or that the architecture is already proven. Right now I’m mainly interested in training behavior, efficiency, scaling characteristics, and how the event-driven approach behaves as the model gets larger.

I’d be interested to hear what people working on alternative architectures or efficient training think about this direction.

https://preview.redd.it/by5k3tlfz7lh1.png?width=1080&format=png&auto=webp&s=03c63a69bf968034bc96727b10e6ac0c79b6ab31

https://preview.redd.it/ewmjr0zfz7lh1.png?width=1080&format=png&auto=webp&s=bb91aa193608bb652eceae480a2cd1bb29a1599a