What unique, custom QOL upgrades have you given your local agents?

Warning: Kinda long post. If you don't like reading, please skip for your own sanity. Also, I've got nothing to sell, just a tinkerer, so I just want to share ideas and learn from you guys too.

When I recently started my agentic journey, I hit a number of walls, the first being tool use eating up way too much context. Also, LM Studio had super slow prompt processing speeds w/Qwen 3.6 27B. I ended up switching to llama.cpp, and then started reading up on harnesses like pi, hermes agent, open code, etc. Eventually, I decided on building my own framework/harness around it so I could have more freedom.

It's been rough; I clearly took the hard path going the custom route, but I prefer the hands-on approach, which has taught me a lot and now I wouldn't have traded the experience for anything, considering how much I've learned.

MCP Broker. I run a LOT of mcp tools. It was eating up my context just loading them, over 20K tokens on startup. The broker hides the tools behind a proxy, and then uses a single mcp tool to run the others. That eliminated the 20K load on startup, and the tool can then read/run the other mcp tools. It's super cool and it works fast.

Temporal awareness. My agent knows the date, time, session duration, context length, the model it's running on, the stack its running on (llama.cpp vs lm studio), how many tokens it's currently using, etc.

Context system warnings. My agent is already aware of its context size, but it also gets system warnings when it hits 85% and another one at 95%:

https://preview.redd.it/ffgdfcplfvih1.png?width=1245&format=png&auto=webp&s=6f757dacbc9ea716707e89e96ede01b969b24641

This is super useful in practice because it triggers the agent to write session summaries on its own, I never have to tell it to do so. It even will suggest I switch models, which moves me to the next feature.

Auto-swap (models). Llama-server has a router mode, so you can load multiple "backup" models. So, when my context limit is nearly hit, my agent will recommend I switch it, or it will switch it itself when it hits 85%. Why? I like to run Q5 at kv 8/8 until 115k to keep speeds fast. At 85-95%, I can auto-swap to Q5 kv 4/4 at 200K ctx and keep chugging along.

Memory search. I made a custom mcp designed specifically for searching memory. It does hybrid search (semantic and vector) via postgres, and I have all the bells and whistles - cosine similarity/distance, pgvector, hnsw. I didn't know what any of this stuff was a couple of months ago.

Memory search is indispensible. I honestly don't know how people can use agents without it. If you set up your memory system robust enough, you can do needle in haystack searches across weeks/months of content.

SIde tip: Mmproj in CPU. If you're not doing this already, you're leaving vram on the table. Running it in cpu, you can shave off 1.7-2GB in GPU, giving your model more room. You'll thank me later.

These are just a few of the customizations I've done (got some wild things I'm working on atm), and it's really thanks to doing things the "hard way", so I'd definitely encourage anyone to try learning from scratch. I'm told my frankenstein system is already far more advanced than hermes/pi/etc. so yeah, I guess that's cool.

Would love to hear what tweaks you guys are doing to your agents. Maybe we can share notes/ideas.