I trained my own 150M non-Transformer language model from scratch on 300M tokens — WarpState
热度趋势
趋势数据积累中
百分比基于当前可用热度信号，而非评论数或独立用户人数。
OpenAI 相关模型动态已经出现，适合跟踪能力变化、生态影响和后续可用性。
一位开发者从零开始训练了一个名为WarpState的150M非Transformer语言模型，使用了300M tokens。训练过程包括9,156个优化器步骤，序列长度为1,024，词汇量为32,768。该模型最终验证损失为3.4309，困惑度为30.90，峰值VRAM使用量约为4.52 GB。开发者正在寻求关于该架构的反馈，特别是对内存更新或共享核心设计的批评。
I’ve been experimenting with alternative language-model architectures for a while, and I recently finished the first complete pretraining run of a new architecture I’m calling WarpState.
This is still an experimental proof of concept, not a claim that it beats Transformers or existing state-space models.
The model has 150.13M parameters and was trained from scratch on roughly 300 million English tokens from Ultra-FineWeb L2.
Parameters: 150.13M Training tokens: ~300.02M Optimizer steps: 9,156 Sequence length: 1,024 Vocabulary: 32,768 Peak VRAM: ~4.52 GB Final sampled validation: Loss: 3.4309 Perplexity: 30.90
https://preview.redd.it/3rmrnnuqq6mh1.png?width=2258&format=png&auto=webp&s=9474446fae97fe506cba2b5b1e308afdd22e6a19
https://preview.redd.it/tuh85jtlq6mh1.png?width=1280&format=png&auto=webp&s=39b0545c50e1dd31e61441ac024e7eba2ed66642
https://preview.redd.it/3txh4lh2r6mh1.png?width=2490&format=png&auto=webp&s=6b271d1e155026b0116171a6cfe54a7b50fd867b
Instead of global self-attention across the entire sequence, tokens are divided into fixed 128-token chunks.
All chunks can be processed as a large batched GPU workload during training, rather than running attention token by token.
The fast state is initialized with a relatively short memory timescale, while the slow state is initialized to retain information much longer.
Fast = decay_fast × Fast + (1 - decay_fast) × write Slow = decay_slow × Slow + (1 - decay_slow) × write
For every token, the model produces a gate deciding how much information should come from:
So the model can use precise local token relationships while relying on the compressed state for information from previous chunks.
Another unusual part of WarpState is that it does not have 16 completely separate large layers.
The current model contains only 4 physical WarpState cores, but they are reused across 16 logical depth passes:
Core 0 Core 1 Core 2 Core 3 Core 0 Core 1 Core 2 Core 3 ...
Each logical depth has a small learned scale and bias, so the same physical core can behave somewhat differently depending on which depth pass it is being used for.
The intention is to get deeper iterative computation without duplicating every large weight matrix.
During autoregressive generation, every logical depth also receives its own independent memory cache, even when two depths share the same physical core weights.
d_model: 1280 heads: 20 head_dim: 64 physical cores: 4 logical depth: 16 FFN hidden: 4480 chunk size: 128 RMSNorm SwiGLU RoPE inside each local chunk tied input/output embeddings
The part I was most interested in was simply whether this architecture could survive a real pretraining run.
I trained it through the full ~300M-token run without NaNs, gradient collapse, or an obvious optimization failure.
Later checkpoints started forming recognizable semantic clusters and reasonably structured paragraphs.
online platform social media sharing content sharing information interaction with other people community