LLMs: Intelligence vs. Cost

-

Staff Software Engineer at OpenTeams. Dask maintainer.

ArtificialAnalysis is a website that benchmarks the intelligence of various LLM models. They publish a headline Intelligence Index , which is calculated as the mean output of the curated selection of benchmarks they run on each model. It’s a decent finger-in-the-air measure of how smart a model is overall.

AA also records useful information — namely, how much it cost them to run the benchmarks. Since the benchmarks are the same across all models, this offers a good indicator of how much it will cost a user to run each model, in relative terms.

One of their main plots is the Intelligence vs. cost plot , which shows the Pareto frontier, i.e. the cheapest model that can achieve each intelligence score. This frontier is important, because using a super-intelligent and super-expensive model to accomplish menial tasks that could be done by a much dumber and cheaper one is just a waste of money.

Over time, I’ve become progressively more irritated by this plot, for a few reasons.

Why AA’s plot is misleading

The first issue I have with it is that it uses a logarithmic scale on the cost axis. Using a log scale is the only way to make you spot the difference between a model that costs $0.015 per task and one that costs $0.032, while the same plot contains a model that costs $3.69 — almost 250 times as expensive. However, the net result is that the viewers can no longer appreciate the immensity of the price difference between the cheap models and the heavy ones; nor can they realize how inconsequential the price differences are between the cheap models.

The second thing that irks me is that it uses the official pricing from the model developers’ own API offering. This is fine in most cases, but for open-weights models it can be a lot more expensive than what the exact same model can be rented for from third-party API providers. OpenRouter makes it very easy to switch providers on the fly and always get the cheapest offer.

The third and final issue is that local models — those that can fit on consumer hardware — appear on the plot at their datacenter pricing, which is always very expensive in proportion to the intelligence you buy with it and ultimately not something any real user will actively want to buy.

I made my own plots

All intelligence index scores are from ArtificialAnalysis. All points are benchmarked at maximum thinking effort where not explicitly stated otherwise. All cost scores are from ArtificialAnalysis too, except where noted below.

In the first plot we see the offering, as of September 1 2026, with the most intelligent (and expensive) models.

A good rule of thumb for reading the intelligence axis: a one-point difference is unlikely to be noticeable by most, while a 5-point gap is substantial. It’s important to point out that an intelligence score of 50, which is the rock bottom in this first plot, is roughly what the smartest model in the world could deliver in February 2026 (Opus 4.6).

The green area at the bottom left is where models become extremely cheap. Let’s zoom into it and extend the intelligence plot a bit lower, down to what can run today on a smartphone.

Some models are marked with a ⚡ symbol. It means that the cost was calculated as the electricity to run the model locally (details on the calculation below), since the model is so small that it makes no sense to serve it from a datacenter. When comparing local models against each other, it also offers a scale of how long each model takes to complete tasks.

Finally, let’s merge the two plots together to better visualize the diminishing returns in performance/cost. Again, the area that’s common to all plots is highlighted in green:

All the differences between AA’s plot and mine

- Changed x scale from logarithmic to linear, because people’s money is not logarithmic

- Changed Kimi K3, Qwen3.8 Max, DeepSeek V4 Flash 0731, GLM-5.3, GLM-5.3-Flash, and Hy3 to the price you can get them for on OpenRouter (excessively slow or unreliable providers and those without Zero Data Retention policies are excluded)

- Extrapolated points for GLM-5.3-Flash at high reasoning effort, by crossing AA scores at max effort with Z.ai’s coding scores at different effort levels

- Changed sub-35-billion-parameter models from datacenter pricing to cost to run locally (read below)

- Added Ornith-1.5-35B-A3B. The intelligence score is extrapolated from self-reported benchmark results by the model authors and should be taken with a healthy dose of skepticism.

Cost calculation for local models