Competence Gate: gating tool-use on a small model's internal confidence signal instead of its verbalised one — Qwen3.5-4B, open weights [P]
这条记录涉及生成能力或端侧推理进展，适合跟踪模型效率、部署门槛和应用机会。
一个为Qwen3.5-4B设计的10MB LoRA适配器，通过读取模型内部激活信号来判断其真实置信度，从而决定是直接回答、网络搜索还是本地文档检索。该工具能更有效地识别错误答案，将错误率降低了0.46，并减少了私密查询泄露到公共搜索的风险，将私密问题发送到公共搜索的比例从22%降至10%。它确保每个答案都可追溯，并能明确表示“我不知道”，而非编造信息。…
I made a 10MB LoRA adapter for Qwen3.5-4B plus a small orchestration layer. It decides, per query, whether to answer directly, search the web, or retrieve from your own local documents and it refuses to make things up when it can't verify an answer.
It runs locally (Apple Silicon / MLX, with a GGUF build for llama.cpp/Ollama).
Basically small instruct models are poor at telling users how confident they really are. They can't verbalise it and tend to say they are confident for everyhting. In my past research I tested seven 3-9b models and they all hit a confidence ceiling. But the information is there in the internal activations. The adapter reads the internal signal directly and gates tool use on it.
The main elements are that:
- it catches its own errors better than the base model's tool calling (d′ improvement of 0.46 (95% CI [0.01, 0.89])). Of the cases the gate flagged that the base model didn't, 87% were genuinely wrong answers.
- it is less likely to leak your private queries to public search. A two-signal version routes personal information related questions such as "what did my discharge summary say" to a local retriever instead of a websearch. It cut the rate of private questions sent to public search from 22% to 10% (reduction 0.12, 95% CI [0.02, 0.22]). This is useful for those who are using the LLM for confidential docs.
- every answer is traceable. When it retrieves, it cites the specific passage ( report.md ¶2
), verifies the answer is actually in that passage, and shows a confidence band. Worst case, it says "I couldn't verify that". It is built to say "I don't know," instead of lie.
limitations:
- Privacy result is n=60; the retrieval/competence dissociation is n=126 hand-authored items. Screened and CI'd, but small.
- GGUF reproduces the MLX gate's decisions at --lora-scaled ...:8
(found by sweep — scale 1 does nothing; effective scale ≈ the training scale). Agreement 0.83 on a 24-item probe; disagreements are all conservative-direction (GGUF answers a couple of borderline items MLX would look up), and knowns never false-fire. Faithful on the safety-critical directions, marginally more conservative at the margin.
- Serve-time confidence is coarse (grounded / declined / answered) — the distilled gate reads nothing at inference, so finer bands need probe access (offline).
- Inherits Qwen3.5-4B's knowledge and biases. The gate governs when to trust the model, not what it knows.
The approach isn't Qwen-specific — I started on SmolLM3-3B, and it should extend to other models and larger sizes.
Repo (weights + code + model card): https://huggingface.co/synthiumjp/competence-gate-qwen3.5-4b
Apache-2.0. It's an open research release. I hope people might find some use for it. Methodology and papers are cited in the model card. Genuinely interested in critique, it's screened work, so if there are any issues it be great to know.
**** Update *\***
I ran the gate against external benchmarks it hadn't been tested on, and one use case did not survive. The gate does not improve grounded document QA — answering faithfully from a provided passage and abstaining when the passage doesn't support an answer. On SQuAD 2.0 unanswerables, fabrication was actually higher with the gate than without it.
The reason is a example of construct specificity. "Knowing when to defer" is not one capability. There are at least two distinct signals hiding inside it:
- Parametric competence: do I know this from my own weights? The gate reads this. It's what the probe was validated against.
- Evidential grounding: is this answer supported by the passage in front of me? A different question, from a different information source.
A probe validated for one carries no usable signal for the other. A parametric-competence signal applied to an evidential-grounding task doesn't just fail to help, it actually interferes by pushing toward answering and suppressing the base model's (Qwen's) own abstention. The base model already handles the easy case (0% fabrication when the passage plainly lacks the answer). The hard case (adversarial unanswerables) needs purpose-built grounded-abstention training, not a post-hoc firewall.
The release is scoped to what's validated: parametric tool-call routing and privacy-aware retrieval routing. The "refuses to fabricate about documents" framing in the original post above is the part that doesn't hold.