I benchmarked Jev aginst gpt-5.6-luna!
A recent benchmark compared TypeSafe's Jev model against gpt-5.6-luna, highlighting Jev's unique design as a non-text-generating model that provides probabilistic answers to typed questions. Jev demonstrated strong reasoning capabilities, outperforming gpt-5.6-luna on benchmarks like LogiQA (0.77 vs 0.59), WinoGrande (0.89 vs 0.66), ARC-Challenge (0.97 vs 0.87), and MMLU (0.94 vs 0.87). Furthermore, Jev achieved a 0.75 score on 120 new math word problems generated with Fable 5.1, comparable to its 0.72 on GSM8K, while gpt-5.6-luna scored 0.17 on the same problems with reasoning off.
Time & source
Times shown in UTC
Display time zone: UTC
Local time zone unavailable; showing UTC.
PublishedOffset at this time: UTC+0Sep 19, 2026, 08:25 UTC
IngestedOffset at this time: UTC+0Sep 19, 2026, 14:00 UTC
- Published
- Sep 19, 2026, 08:25
- Ingested
- Sep 19, 2026, 14:00
- Source type
- Dev community
- Tier
- Community
- Source status
- Healthy
Tier is a per-source editorial setting, not a per-item score.
I got access to TypeSafe's Jev a few days ago. It's an odd kind of model that doesn't generate text at all. You send it some content plus typed questions (yes/no, pick one of these options, rate this on a scale) and it gives you back probabilities.
Setup: 49 tasks, about 8,200 items, all from public labelled datasets (SST-2, MMLU, ARC, MS MARCO, XNLI, SQuAD, Banking77 and so on), plus some synthetic tests where code computes the right answer. Both models got the same question wording. The baseline was gpt-5.6-luna with reasoning off, and again with reasoning on low.
~105ms median server time vs 700-800ms about $0.04 per 1,000 items vs $0.16-0.19 calibration error roughly half the baseline's, so the probabilities actually mean something
The part that surprised me was reasoning. LogiQA 0.77 vs 0.59, WinoGrande 0.89 vs 0.66, ARC-Challenge 0.97 vs 0.87, MMLU 0.94 vs 0.87. I assumed it had memorised the benchmarks, so I generated 120 new math word problems with a script using Fable 5.1. It got 0.75 on those, about the same as its 0.72 on GSM8K. Luna with reasoning off got 0.17 on the same problems.
Reranking was also a clear win (NFCorpus nDCG@10 0.73 vs 0.63), and Luna needed 1-3 seconds per query there.
Where it lost: counting things in a list (0.87 vs 0.99 once Luna could reason) picking 1 of 77 intents (Banking77, 0.81 vs 0.87) asking a question and then its negation did not give probabilities that add up to 1, off by about 0.3 on average
in a separate private test with ~100 long documents in one request, it gave high relevance scores to a few completely unrelated ones. None of the clean benchmarks show that.
because I know someone will ask. These are public benchmarks, so contamination is possible for any model, and the new math set is my only control for that. The baseline was deliberately held to no/low reasoning. 200 items per task means anything under about 0.05 is noise. Only 7 of the leads are clearly outside the error bars. And it can't write text, call tools or explain itself, so it doesn't replace your LLM. It replaces the small classifier, reranker and "is this relevant" calls around it.
Repo has the task builders, runner, scorer, my raw per-item results and the full table. Python stdlib only. You can plug in any OpenAI-compatible model as another baseline with one argument, along with the full results: