31,352 repeated LLM measurements suggest static benchmarks miss important temporal variation

AI benchmarks answer a snapshot question:

How well did this model perform when it was tested?

But for API-served models, there is another question that matters just as much:

Does the same model continue to behave the same way over time?

We have been continuously evaluating LLMs across coding, multi-turn reasoning, and tool use, and one of the more interesting patterns appeared when we compared variation within a day with variation across days.

In one historical analysis covering 31,352 repeated score observations across 49 models, we observed:

- within-day standard deviation: 2.80 points

- between-day standard deviation of daily medians: 8.43 points

- descriptive ratio: approximately 3.01×

That does not mean providers were changing their models every day.

There are several possible confounders: sampling variation, task composition, missing observations, provider-side serving behavior, infrastructure effects, and changes to the evaluation instrument itself. Daily medians reduce the effect of extreme observations, but they do not eliminate serial dependence or shared day-level effects.

The interesting part is that the variation is large enough that treating a model score as a permanent property of the model starts to look questionable.

This led us to think about benchmarking less like a leaderboard and more like an observability system.

For longitudinal evaluation, we now try to separate several things that can otherwise look identical:

Model behavior changed. The model performs differently on the same type of workload.

The benchmark changed. A prompt, test, scoring rule, or task definition was modified.

The provider changed. Serving conditions, throttling, infrastructure, routing, or backend behavior changed.

Randomness happened. Repeated calls naturally produce different outcomes.

These distinctions matter because a lower score alone is not evidence of model degradation.

One design choice we found particularly important is benchmark versioning. If you change a task, test case, prompt, request policy, or scoring definition, the historical baseline should not silently continue as if the measurement instrument were unchanged.

In other words, the useful question is not:

It is:

We also separate availability problems from capability results. A provider timeout, authentication problem, or transport failure should not automatically become evidence that the model became less capable.

Another difficult issue is benchmark contamination.