Skip to content
RCreddit.com·
Not on the current live radar

Update: Small model + Engram

AI summary

A developer reported surprising coherence from a small 9b model after only 15 million tokens, attributing the success to borrowed embeddings and an LM_head down-projected from 5k to 2048 d_model. This method preserved approximately 65% of the data from a larger model's embedding, which significantly improved results compared to using pure 1 hot cross entropy that would have produced gibberish.

Time & source

Times shown in UTC

Display time zone: UTC

Local time zone unavailable; showing UTC.

IngestedOffset at this time: UTC+0Sep 17, 2026, 15:00 UTC

Ingested
Sep 17, 2026, 15:00
Source type
Dev community

Full text isn't available here.

Read at source →
Source·reddit.com