JEV almost dead: CLM vs JEV

Jev, a proprietary model, maintains an edge in zero-shot open-domain tasks, as evidenced by its 99.2% score on the Berkeley Function Calling Leaderboard v4 compared to CLM-8B's 95.2%, and a perfect 30/30 on WikiRacing versus CLM-8B's 26/30. However, using CLM offers significant latency gains and zero API costs, providing the full primitive set (Choice, Noul, Score) while only sacrificing some zero-shot generalization on niche out-of-domain tasks.

This report contrasts Jev's zero-shot accuracy with CLM's latency gains and zero API costs, unlike typical comparisons that focus solely on performance metrics.