Measuring AI general intelligence instead of just averaging benchmarks

TL;DR: Most aggregate AI leaderboards ask:

“How well did this model score across the benchmarks we chose?”

GII instead asks:

“What underlying level of general capability would most likely produce this entire pattern of benchmark results?”

I think the second question is much closer to what people actually mean when they argue about which model is “smarter.”

AI leaderboards have a surprisingly basic problem: how do you combine a bunch of completely different benchmarks into one meaningful number?

A common solution is basically to average them.

For example, Artificial Analysis’ Intelligence Index aggregates performance across a collection of evaluations into a single index. That's useful for getting a quick overview, but benchmark averages have some important statistical problems.

- Every included benchmark equally affects the final ranking, even if some are much better measures of general capability than others.

- Correlated benchmarks can effectively double-count the same ability. If two tests measure nearly the same thing, doing well on both can have disproportionate influence.

- Benchmark selection matters a lot. Adding/removing tests can change rankings even when the underlying models haven't changed. A benchmark can be very difficult without necessarily being a particularly good measure of general capability.

So I built the General Intelligence Index (GII) around a different idea:

Instead of asking “What is the model's average benchmark score?”, ask “What latent ability best explains its performance across all these different benchmarks?”

It's basically the psychometric approach used to estimate human general intelligence in IQ tests, adapted to AI evaluation. GII currently uses 59 benchmarks and 267 models and estimates a latent general factor ( g ) using multidimensional item-response theory.

The interesting part is that the benchmarks themselves get measurable psychometric properties. For example, the estimated g-loadings range from:

ANLI — .986 EBR-bench — .983 Winogrande — .978 OSWorld 2.0 — .978 ARC-AGI — .958 ARC-AGI-2 — .955 HLE — .952 METR Time Horizons — .935

...all the way down to:

LAMBADA — .506 TriviaQA — .290

This matters because GII doesn't simply treat 90% on Benchmark A as equivalent evidence to 90% on Benchmark B. It considers things like g-loading, difficulty, discriminability, reliability, measurement error, and covariance between benchmarks.

That last part is especially important. Suppose a leaderboard contains five highly correlated coding benchmarks and one reasoning benchmark. An average can effectively give “coding ability” five votes. A latent-variable model can recognize that those five tests contain overlapping information and discount the redundancy.

The final latent scores are normalized onto an Artificial IQ (AIQ) scale: Mean = 100, SD = 15. This isn't supposed to mean “GPT-5.6 literally has a human IQ of X.” It's just a familiar standardized scale for comparing AI systems.

Current top results:

Claude Fable 5.1 — 137 AIQ GPT-6 Astra — 137 Claude Fable 5 — 131 GPT-5.6 Sol — 131 Claude Opus 5 — 130

GII: https://indexgi.vercel.app/