mini-AGI: Continual-learning dynamically looped transformer with evolutionary grown (on a laptop)

A new mini-AGI model, described as a continual-learning dynamically looped transformer, has been developed with evolutionary growth on a laptop. This intriguing design incorporates a Mixture of Experts architecture, featuring 8 active experts, 32 routed experts held in VRAM, and smart caching for 96 additional experts. The community finds its unique approach and design choices noteworthy.

This report highlights a mini-AGI model that uniquely combines continual learning with evolutionary growth on a laptop, unlike larger models requiring extensive data centers.