I tested Anthropic’s new Jacobian Lens on open models, then it turned into a local-model hallucination router

Anthropic dropped their Global Workspace / Jacobian Lens paper yesterday, and I thought it was too cool not to try on open models.

Normal prompts, emotional prompts, ragebait prompts, deletion-threat prompts, base vs abliterated, small vs bigger models.

- Gemma 4 E4B
 - Gemma 4 12B
 - Gemma 4 12B abliterated
 - Gemma 4 26B MoE
 - Qwen 3.6 27B

When the model knows the answer, the workspace looks calm. One candidate starts winning early, layers mostly agree, and the answer forms cleanly.

When it is about to confidently guess, the workspace looks foggy. Competing candidates stay alive through the middle/deep layers, then the final layer still picks something fluent.

Then I fit a tiny logistic-regression router on workspace trajectory features: entropy slope, late-band entropy, entropy std, answer rank, layer agreement, etc.

E4B: logprob .711 | workspace .773 | combined .787 12B: logprob .736 | workspace .824 | combined .843 12B ablit: logprob .731 | workspace .799 | combined .812 26B MoE: logprob .725 | workspace .749 | combined .783 Qwen 27B: logprob .856 | workspace .646 | combined .838

This works well on the Gemma models. Workspace features beat output confidence alone on every Gemma model I tested.

It does not work universally. Qwen is the miss. Its output confidence is already very well calibrated, and workspace features do not help there.

answer locally take one workspace snapshot tiny router scores risk if confident but foggy, escalate to search, citations, or a bigger cloud model

The trained routers are uploaded too. The E4B router transfers zero-shot to the other Gemmas at about 0.74-0.78 AUC. The whole thing is just a small logistic regression, which is kind of the point.

The biggest E4B router weight is entropy slope. That was interesting to me: the danger sign is not just “foggy,” it is the workspace getting foggier as the model goes deeper.

Side finding: fake entities are a different failure mode. Logprobs catch most fake-entity prompts because the model usually knows the name is unfamiliar.

But abliteration did something wild. The base 12B fabricated on 17/50 fake entities. The abliterated 12B fabricated on 49/50. Same base weights, very different “I don’t know” behavior.

The narrower thing I’m testing is whether Jacobian-lens workspace trajectory features are useful as a one-pass risk signal for confident wrong answers, especially for local-to-cloud routing.

If this exact angle already exists, I’d genuinely love pointers. I want to build on the right prior work, not reinvent it badly.

- real inference overhead in a local serving stack
 - a lightweight router sidecar
 - more model families
 - harder datasets where output confidence is miscalibrated
 - tool-use
 - whether abliterated models lose useful “I don’t know” signals

Also, if anyone is working on model honesty, evals, interpretability, or local-to-cloud routing, I’d be happy to talk. This is exactly the kind of work I want to do.

EDIT: I'm going to keep running experiments until I run out of compute. Next up:

- deliberately vague inputs, so I can tell a well-judged guess from a lucky one

- agent traces and tool calls, like does it get foggy right before it invents a tool that doesn't exist

And if you think I don't know what I'm doing, you're probably right, just correct me if you feel like it. I just love doing this stuff.