Running a vision + audio + reasoning on one Gemma 4 E2B locally on 4 GB VRAM — and keeping it real time.

So I've had one Gemma 4 E2B running through llama-server as the only model in a local tool that watches my screen and lets me search/chat over it later. Same model does all three jobs:

- looks at the screen and turns it into structured info (what app, what I'm doing, rough layout)

- audio — voice memos + meeting transcription using E2B's audio encoder, so I didn't have to bolt on Whisper

- the actual chat/RAG over history(you can try chatting with a rag built over your screen history)+ daily summaries

Since it's one model on one GPU, everything's fighting for the VRAM (I built this on a 4GB GTX 1650, so not a lot to go around). Honestly most of the effort didn't go into the AI part, it went into optimization since i wanted it to be more of a background service

-Chat interrupts screen analysis. llama-server runs with --parallel 1 (single slot — on 4GB I'd rather have one good response than two slow ones). If I send a chat message while it's mid-way through analyzing a screenshot, it kills that in-flight request — closing the HTTP connection makes llama-server drop the slot in under a second — waits for the slot to actually free, then answers. The analysis that got killed goes back to the front of the queue so nothing's lost.

-Perceptual-hash cache so it's not re-analyzing the same screen repeatedly. Before it calls the model it pHashes the frame and compares to the last one it did for that same app+window. Basically identical -> reuse the old result, no model call. Changed a bit -> reuse the layout, quick pass. Actually different -> full pipeline. Chat apps go stale faster than my editor,so they get a shorter window. Day to day this skips most of the calls.

"fast" mode prefills the assistant with an empty to skip the reasoning tokens, OCR (easyocr) gets fed in as text so the model doesn't burn reading every single text, screenshots get shrunk to 768px to fit, embeddings run on MiniLM on CPU so they never touch the GPU, and capture just pauses when something like a game or video editor is in focus.

fast mode: ~12s/frame on the 1650 (model spills into system RAM), ~3-4s on a 3060 once the whole thing fits in VRAM (that's the real jump,), ~1s on a 4090.(my approximations)

The tool's open source, called ScreenMind. I put out a really rough version here a while back and it's come a long way since — it's pip install screenmind now, has an in-app model hub so you're not in the terminal to download/switch models, does multi-model, and runs on Linux/Wayland too. got ~170 stars. Also exposes everything to Claude/Cursor over MCP.

(people always go "isn't this just Recall" — difference is Recall/screenpipe mostly dump raw OCR text, here the model actually reads the frame and OCR is just context.)

Privacy was also what i thought off since it's literally watching your screen: 100% local, zero network calls after the model download, no telemetry ever. Screenshots are encrypted at rest, and it auto-redacts credit cards / API keys / passwords out of the captured text before anything gets saved. There's also an incognito toggle and a PIN lock on the dashboard.

Anyway, currenly i am working on multi monitor support on it so if anyone got anything in mind..welp!