Pruning RAG context down to what the answer actually needs

Kapa builds AI assistants that answer complex questions over large product knowledge bases. Think technical documentation, API references, PDFs, forums, support threads. Developers use our retrieval API to give their agents context about their product, and the same retrieval layer powers our end-to-end assistants.

For all the debate in 2026 about whether agents still need RAG, in our domain nothing comes close when knowledge bases get large and complex. Our retrieval comes in several forms, some agentic, some single-pass, but they all share the same shape: a retriever, which finds the chunks of documentation relevant to a question, and a generator, the LLM that writes the answer from them.

The short version of this post: we added a third step between the two. A small, cheap LLM reads the question and all the retrieved chunks together, and throws out the chunks the answer will not need before the expensive model ever sees them. It drops about 68% of the context, keeps about 96% of recall, and cuts the cost of a query by a third, net of its own cost. This post explains how we got there.

**Ignored chunks still cost money**

A retriever is a funnel. Embedding and keyword search cut a knowledge base of hundreds of thousands of chunks down to a few hundred candidates, a reranker orders them, and the top 15 or so reach the generator, the largest and most expensive model in the chain. Even then, most of what the generator reads is not needed for the question. That is deliberate: retrievers aim for maximum recall and trust the generator to ignore the noise.

But the generator is billed for every chunk it ignores. In our assistants, retrieved chunks are about two-thirds of the cost of a query, more than the answer, the conversation history, and the system prompt combined. Every chunk fewer cuts the query cost by about 4%. And in an agent, every tool call pours its output into the same context, so the context grows quickly; a tighter retrieval result buys room for everything else the agent has to hold and leaves less context to rot.

The catch is recall. Drop a chunk the answer needed and you traded a few cents for a wrong answer. A pruner is exactly as good as that tradeoff: compression gained per point of recall lost.

**The obvious fix does not work**

We already rerank before returning the top K, so we are sometimes asked to just expose the rerank scores and let callers cut on them: keep everything above 0.7, drop the rest. It fails for two reasons, and the second shaped everything we built.

First, a rerank score is an ordering, not a measurement. It says chunk A beats chunk B on this query, nothing more. The scores are not calibrated across queries, Cohere too says as much, so no fixed cutoff works. The only cutoff a ranking supports is positional, top-N, and that drops the last chunk whether it is noise or the answer.

Second, and this survives even perfect calibration: relevance is not a property of a single chunk. The rerankers in most pipelines are pointwise cross-encoders. They score each query-chunk pair alone, never alongside the other chunks it was retrieved with. Here is an anonymized production example:

The second chunk never mentions audit logs, so it scores as noise, yet it is half the answer, and no pointwise score can see that, because the chunk is only relevant next to the first one. Chunks also split multi-part questions between them, each useless alone. The real question is never whether a chunk is relevant by itself, but whether it belongs to a set that together answers the question.

**A clever fix that fails the same way**

Before giving up on the reranker we tried anchor documents ( Sinhababu et al. ): make the reranker's scale absolute by planting synthetic chunks of known relevance into the ranking, one written per level from Essential to Unrelated, then drop every real chunk that ranks below the anchor of the lowest level you want to keep. One extra LLM call on top of a rerank you already run, and genuinely elegant.

It did not work, for the same underlying reason. Anchors fix calibration, but they cannot fix the scores, and the reranker kept placing partially and indirectly relevant chunks below plainly irrelevant ones. To keep them, the anchor has to sit so low that hardly anything gets pruned.

That failure was the useful result: whatever prunes has to see the question and all the chunks at once, because the thing being judged is the set.

**So we let an LLM grade the chunks**

What we shipped is one listwise LLM call between the reranker and the generator. It gets the question and all the chunks, and grades every chunk against a five-level scale written into its prompt:

LLM Score

Level

Meaning

5

ESSENTIAL

The answer cannot be produced without this chunk, whether it answers directly or is a definition or prerequisite another chunk depends on.