I built an open-source MCP server that gives AI agents persistent memory for videos
这条记录涉及编程工具或代码能力更新，适合开发者评估工作流变化和可复用价值。
开发者创建了一个名为 Watch Skill 的开源项目，旨在解决大型语言模型处理视频时缺乏持久记忆的问题。传统工作流程中，视频被视为临时输入，导致每次提问都需要重新上传和分析。Watch Skill 通过在首次分析时创建包含转录、OCR、视觉观察、时间戳和嵌入的持久本地索引，将后续问题转化为检索，而非重复的视频处理。…
Disclosure: I'm the developer of this project.
Over the last few months I kept running into the same workflow problem.
LLMs have become very good at reasoning over text. We have RAG pipelines, long-term memory, vector databases, and increasingly capable agents that can search documentation, code, and previous conversations.
Videos felt completely different.
A typical workflow looked like this:
- Upload a bug recording or product demo.
- Ask a few questions.
- End the session.
- Upload the same video again the next day.
The model wasn't the problem. The workflow was.
So I built an open-source project called Watch Skill to experiment with a different approach.
Instead of treating a video as temporary input, the first analysis creates a persistent local index containing transcripts, OCR, visual observations, timestamps, and embeddings.
Future questions become retrieval instead of video processing.
The project exposes the same functionality through MCP, a CLI, and a REST API, so it isn't tied to one specific agent. It also supports running fully offline with local models if that's important for privacy or cost.
Some implementation decisions that ended up mattering more than I expected:
- Scene detection instead of uniform frame extraction.
- Hybrid retrieval (FTS + embeddings) instead of vector search alone.
- Timestamped evidence for every answer.
- Persistent indexing so follow-up questions don't require another full video analysis.
- A local correction system where mistakes can become reusable lessons for future queries
I'm posting this mainly because I'm interested in feedback on the architecture rather than promotion.
If you're building agent systems that work with video, would you solve this differently?
Repository:
https://github.com/oxbshw/watch-skill