I trained my own 150M non-Transformer language model from scratch on 300M tokens — WarpState
Heat trend
Collecting trend data
The percentage is based on available heat signal, not comment count or independent people.
OpenAI model activity is surfacing — worth tracking for capability changes, ecosystem impact, and availability.
A developer trained a 150M non-Transformer language model from scratch, named WarpState, using 300M tokens. The training involved 9,156 optimizer steps with a sequence length of 1,024 and a vocabulary of 32,768. It achieved a final validation loss of 3.4309 and a perplexity of 30.90, with a peak VRAM usage of 4.52 GB. Feedback is sought on the architecture, particularly regarding the memory update or shared-core design.
I’ve been experimenting with alternative language-model architectures for a while, and I recently finished the first complete pretraining run of a new architecture I’m calling WarpState.
This is still an experimental proof of concept, not a claim that it beats Transformers or existing state-space models.
The model has 150.13M parameters and was trained from scratch on roughly 300 million English tokens from Ultra-FineWeb L2.
Parameters: 150.13M Training tokens: ~300.02M Optimizer steps: 9,156 Sequence length: 1,024 Vocabulary: 32,768 Peak VRAM: ~4.52 GB Final sampled validation: Loss: 3.4309 Perplexity: 30.90
https://preview.redd.it/3rmrnnuqq6mh1.png?width=2258&format=png&auto=webp&s=9474446fae97fe506cba2b5b1e308afdd22e6a19
https://preview.redd.it/tuh85jtlq6mh1.png?width=1280&format=png&auto=webp&s=39b0545c50e1dd31e61441ac024e7eba2ed66642
https://preview.redd.it/3txh4lh2r6mh1.png?width=2490&format=png&auto=webp&s=6b271d1e155026b0116171a6cfe54a7b50fd867b
Instead of global self-attention across the entire sequence, tokens are divided into fixed 128-token chunks.
All chunks can be processed as a large batched GPU workload during training, rather than running attention token by token.
The fast state is initialized with a relatively short memory timescale, while the slow state is initialized to retain information much longer.
Fast = decay_fast × Fast + (1 - decay_fast) × write Slow = decay_slow × Slow + (1 - decay_slow) × write
For every token, the model produces a gate deciding how much information should come from:
So the model can use precise local token relationships while relying on the compressed state for information from previous chunks.
Another unusual part of WarpState is that it does not have 16 completely separate large layers.
The current model contains only 4 physical WarpState cores, but they are reused across 16 logical depth passes:
Core 0 Core 1 Core 2 Core 3 Core 0 Core 1 Core 2 Core 3 ...
Each logical depth has a small learned scale and bias, so the same physical core can behave somewhat differently depending on which depth pass it is being used for.
The intention is to get deeper iterative computation without duplicating every large weight matrix.
During autoregressive generation, every logical depth also receives its own independent memory cache, even when two depths share the same physical core weights.
d_model: 1280 heads: 20 head_dim: 64 physical cores: 4 logical depth: 16 FFN hidden: 4480 chunk size: 128 RMSNorm SwiGLU RoPE inside each local chunk tied input/output embeddings
The part I was most interested in was simply whether this architecture could survive a real pretraining run.
I trained it through the full ~300M-token run without NaNs, gradient collapse, or an obvious optimization failure.
Later checkpoints started forming recognizable semantic clusters and reasonably structured paragraphs.
online platform social media sharing content sharing information interaction with other people community