A trained fast-weight memory: a 3M-param transformer installs never-trained rules at inference, forward-only — where test-time training transfers nothing (single RTX 3090, fully reproducible)

I'm an independent researcher (single self-funded RTX 3090). I just released a preprint (Zenodo for now — arXiv pending endorsement) on training a fast-weight memory bank: a small bank of vectors that the model writes with its own forward pass and reads as weights (each slot is expanded by a hypernetwork into a low-rank MLP layer applied to the token stream) — not attended as data. The goal is continual learning at inference without any backward pass: no TTT, no optimizer, no weight clone, no growing context.

Setup: a 3.08M-parameter DeepSeek-style transformer with an 8-slot bank, on a keyed multi-turn rule task — each conversation binds K=2 key tokens to fresh modular rules, presents each rule once (13 tokens), then queries unseen symbols on later turns. Each turn is a separate forward pass, so the rule can only cross turn boundaries through the bank. Chance is 0.008, and bank ablation is an exact control (it sits at chance everywhere).

- It works and generalizes. A single 13-token presentation installs a never-trained rule at 0.79–1.00 accuracy on unseen queries (two seeds). The rule survives physical eviction of its slot (storage turns out to be a redundant superposition — evicting a slot removes a copy, not the content), and can be replaced mid-conversation in one forward pass with old-rule persistence exactly 0.000.

- It's the only pathway that works. Head-to-head on the same conversations: test-time training with a full LR × steps sweep fits its adaptation examples (0.99) and transfers exactly nothing to unseen queries — at 138× the cost per rule update, and destroying 62% of a concurrent untouched rule (the bank loses 14%, by eviction pressure). In-window ICL is also at chance.

- Memory policy is trained, not architectural. The same architecture trained on fixed-structure conversations perseverates totally on a rule switch, zero-shot (old-rule persistence 1.000 — it cannot even produce a readable write on a dirty bank). Randomizing conversation structure at training time (lengths, switch positions) installs the full keep/overwrite/write-on-dirty policy. What the memory does is decided by the training distribution.

The honest caveats, because they're half the paper: this is deliberately small-scale and synthetic — the controls (exact ablation, cost accounting, held-out rules) are the point, and they would blur at scale. Training the read/write circuit is not free: the joint gradient has an ignore-the-bank fixed point, and breaking it needs a teacher-forced bootstrap + annealing + a rule-diversity threshold (below ~112 training rules, held-out accuracy is exactly 0.000 — the read memorizes). A never-trained rule family defeats the bank, TTT and ICL equally: the boundary is the meta-training envelope, not the mechanism. And the replacement policy bifurcates across seeds (selective update vs flush-and-rewrite).

Reproducibility: everything (3 training runs ~5h each on one 3090, probes, figures) reproduces from a fresh clone with one script.

Transparency note: the experimental campaign and drafting were done with Claude (Anthropic's Fable model) via Claude Code, with human scientific direction — the paper is explicit about this split.

Happy to answer questions — especially skeptical ones about the TTT comparison, which is the part I most wanted to get right.