Lesson learned. Don't blindly trust repos and make sure everything is stable for a long running (multi weeks) benchmark.

A developer shared a lesson learned about ensuring stability in long-running benchmarks, specifically for a Django 100 tasks benchmark comparing local models and quantization. They discovered an instability in their evaluation workflow, leading to a re-evaluation of all runs. Key findings from the corrected evaluation include Flash Next remaining the top performer, with the benefit of xhigh versus medium reasoning effort now accurately reflected. Similar observations were made for 3.8 27B, though medium reasoning is still preferred for everyday tasks.

This report highlights how an unstable evaluation workflow led to inaccurate benchmark results, unlike the initial findings, and now properly shows the benefit of xhigh versus medium reasoning effort.