I accidentally turned LLM memory into program analysis

Over the past few months I have been playing around quite a bit with LLM agents, particularly for vulnerability research.

They are becoming surprisingly good at navigating large codebases, explaining unfamiliar subsystems and helping explore potential attack surfaces. However, once an investigation starts taking a few hours, I kept running into the same problem: the model would slowly lose track of what we had actually established.

It might suggest an approach that we had already ruled out, forget that an assumption turned out to be false, or confidently continue reasoning from an observation that was no longer valid. Obviously, telling an LLM that something is wrong does not necessarily mean that it will stop believing all of the things that depended on it :)

I initially started looking into memory systems because I wanted to make LLMs more useful for complex vulnerability research and reduce this type of hallucination.

There are of course already plenty of solutions for giving LLMs memory. Usually this involves storing old conversations or observations somewhere, embedding them, and then retrieving the most relevant pieces whenever the model needs them again.

This works reasonably well, but there was something about it that bothered me.

During a vulnerability research sesh, I don’t just want the model to remember what we said.

I want it to maintain what we currently know.

Imagine that during an investigation we establish the following:

attacker controls object_a object_a points to object_b object_b is a kernel object

From this, we may conclude that the attacker can control a kernel object.

A normal memory system could store all of these observations and retrieve them again whenever we ask about the exploitability of the bug. The LLM then figures out the same conclusion.

Great!

However, suppose that two hours later we discover in LLDB that object_a does not actually point to object_b, and that our previous observation was based on a wrong assumption.

At that point our memory may contain something like:

object_a points to object_b attacker can control object_b object_a does not actually point to object_b

Now we retrieve some subset of these memories and hope that the LLM correctly figures out which conclusions are still valid.

This started to feel a little familiar to me.

This looks like program analysis

A lot of the work I normally do involves program analysis.

When analysing a program, we usually have a bunch of facts about the program and some rules that derive additional facts from them.

For example, imagine we know:

calls(foo, bar) calls(bar, baz)

We could define a rule stating that if one function calls another function, which itself can reach a third function, then the first function can reach the third function as well.