Tauon: A new optimizer outperforming Muon on GPT-Mini (lower loss, ~8.5% faster step time) [P]

A new optimizer called Tauon has been introduced, demonstrating superior performance over Muon and AdamW on GPT-Mini. Tauon achieved a lower final validation loss of approximately 1.6, compared to Muon's 1.65 and AdamW's 1.8. Additionally, Tauon exhibited an approximately 8.5% faster step time, indicating improved efficiency in its operation.

This report introduces Tauon, a new optimizer that not only achieves lower validation loss than Muon and AdamW on GPT-Mini but also boasts an 8.5% faster step time.