I analyzed 31,352 hourly LLM benchmark scores: within-day variation was 2.8 points, while between-day variation was 8.4 [P]

https://preview.redd.it/42s57e5oqamh1.png?width=1903&format=png&auto=webp&s=69958a72e22276534b3605d11f3e1721f76e59c9

Disclosure: I developed AIStupidLevel, the open-source system used to collect and analyze this data. Both the frontend and backend are MIT-licensed.

Most LLM evaluations measure performance at a single point in time. I wanted to investigate a different question:

How stable are the models behind production APIs over time, and how can sustained performance changes be separated from ordinary stochastic variation?

I built a continuous evaluation pipeline that repeatedly tests models across coding, deep reasoning, tool calling and high-frequency canary tasks.

Coding responses are executed rather than judged only through model-based evaluation. Tool-calling tests require models to select tools, construct valid arguments and complete workflows inside isolated Docker environments.

Tasks are executed five times, with the results aggregated to reduce the influence of unusually strong or weak generations. Prompts, scoring logic and API parameters are kept consistent wherever supported by the provider.

This suggests that isolated hourly movements are dominated by normal model stochasticity, while sustained changes across daily evaluation windows provide a materially stronger signal for detecting performance drift.

The detection pipeline aggregates repeated measurements into daily medians and applies sequential change-point detection. Potential incidents must persist beyond the expected historical variance and pass statistical and minimum-effect thresholds before being classified as degradation or recovery.

This analysis became the foundation of AIStupidLevel, a continuous LLM benchmarking and drift-detection system.

The attached screenshot shows the live monitoring interface. Models are classified as stable, volatile, degraded or recovering, with separate measurements for coding, reasoning, tool use, reliability, latency and price.

At the time of the screenshot, the system detected a 32% sustained performance decline in Gemini 3.1 Flash Lite and classified it as a critical incident.

The same continuously updated dataset also powers an OpenAI-compatible router that selects models according to their current task-specific performance, stability, tool-calling reliability, latency and cost.

This adds a missing observability dimension for production LLM systems. Existing monitoring generally captures availability, errors, latency and token cost. Continuous evaluation also measures whether the model remains capable of performing the work for which it was selected.

I would be interested in comparing this approach with other temporal evaluation or change-point detection systems being used for production LLM monitoring.