Introducing Olmo-core 3: Open, scalable training infrastructure for large MoEs

Olmo-core 3 is an open and scalable training infrastructure designed for large Mixture-of-Experts (MoEs) models. Benchmarked on NVIDIA B300 GPUs, it achieved a throughput of 858 TFLOP/s/GPU with a 1.2-trillion-parameter model, using 58.36 billion parameters active per token across 512 GPUs. These benchmarks focused on system performance using random routing, not the quality of a trained model. Further details on its design and experiments are available in its technical report and on GitHub.