Training a 3.8B LLM to 0.384 CORE for $998

A researcher successfully trained a 3.8B LLM to 0.384 CORE for $998, demonstrating that meaningful models can be trained by individuals with limited budgets. The training involved 25,000 steps and 57.3B tokens, taking 35.9 hours. The process achieved a steady state of approximately 480,000 tokens/sec. Initial attempts at document-boundary masking with flex attention were discarded, as cross-document leakage was found not to significantly worsen performance, leading to a simpler best-fit packing approach.

This report uniquely details the specific cost ($998) and performance (0.384 CORE) for training a 3.8B LLM, unlike general discussions of budget-friendly model training.