KV cache as an agent runtime [R]
A research team is exploring using the KV cache as an agent runtime to enhance LLM interactivity and responsiveness. This approach, detailed in previous papers like Hogwild! Inference and AsyncReasoning, involves modifying the model's inference state. Future work includes a Qwen3.8-27B agent interactively playing a DOOM environment using these techniques, suggesting that model inference/runtime design might be an underexplored aspect of agent capabilities.
- Published
- Sep 7, 2026, 09:03
- Source type
- Dev community
- Tier
- Community
- Source status
- Healthy
Times shown in UTC
More details
Our research team has been exploring an alternative approach to achieving interactivity and better responsiveness with LLM systems.
One of the team members wrote up a post about it: https://research.yandex.com/blog/the-kv-cache-as-an-agent-runtime
The post sums up the overall idea of modifying models inference state (KV-cache) for achieving a more interactive LLMs. This idea was used in our lab's previous papers Hogwild! Inference, and AsyncReasoning, the post also contains a preview of the future work in this direction, where a Qwen3.8-27B agent is playing a DOOM env interactively using similar techniques.
We think that its interesting whether model inference/runtime design is itself an under-explored axis of agent capabilities, alongside models and the harness (e.g. harness is too abstract, changing model is too costly, do we need something in between?)