I trained a 3.87B MoE (1.45B active) from scratch on only 86.5B tokens

A developer trained Apex-2, a 3.87B MoE model (1.45B active) from scratch using only 86.5B tokens. The model, with a Decoder-only MoE architecture and 32 layers, achieved a HumanEval+ score of 41.5, matching Qwen2.5-1.5B despite significantly less pretrain data. However, it showed limitations in multilingual ability, knowledge, and math, and DPO training negatively impacted its performance.

This report highlights that Apex-2 matched Qwen2.5-1.5B's HumanEval+ score using only 86.5B pretrain tokens, unlike Qwen2.5-1.5B's 18T tokens.