Show HN: TERMy – A fast terminal assistant that does not use LLMs

NPC-Forge and TERMy development

I love research and development, you may have heard of me because of PJON (Padded Jittering Operative Network) . It is a network protocol I started developing in 2010, which was recently implemented in silicon by the ETH Zurich university thanks to the research of Pius Sieber .

I had a chance to focus for 2 months on my personal projects since early July, during the strange times of AI price hikes and the end of subsidized tokenmaxing. I was curious to see if I could develop from scratch a terminal assistant capable of handling simple natural language requests. I have a bad memory and got used to ask to copilot "activate the virtual environment" or similar trivial operations spending a non negligible sum every month. I started thinking, maybe I can do something to make my workflow more efficient? Do I really need trillions of parameters to accomplish those tasks?

Transformers at home

I started an open-ended research on the feasibility of implementing a generative model at home and training it from scratch on the computer I used to play Kerbal Space Program in the early 2010s "upgraded" with 16GB of RAM, NVIDIA GTX 1050 Ti (4GB VRAM) and a i7-4790K (4.0GHz 8 cores) CPU. In my experiments I tend to look for minimalism, so I imposed myself a constrained environment to be forced to work towards an elegant and efficient solution.

I first developed a framework to train and evaluate transformers, which I implemented from scratch in Python. I have started with something very similar to NanoGPT with 100-200M parameters, then I added flash attention, and all the expected optimizations, I even tried novel architectures like Mamba. The results were generally unsatisfactory, creepy if not outright scary, like the following:

Enter your prompt (or leave blank for empty start):

What is an alien?

Generating...

Using tokenizer: /ollm/checkpoints/gpt/tinyostrich/tokenizer.model

=== Generated Text ===

He's not a member of the world. He can't believe anything anymore. All of those animals are looking like excrements, but every mouth is not a bad one. They look alike, all of which are really terrible. They get their own life, and each is a sign that they're not. They're all right, they say. They're just some different things they can make. They've never seen them. They were better not. Some of them've got their rights. Some of them have to go. But we're not looking for the future on how many. The last two of them are all in the universe.

(the word excrements was used instead of an expletive composed of 4 letters that I am not willing to publish in here)

All my tests produced models that were prone to enter in loops repeating the same sentence, and even if trained on QA they were rarely able to consistently answer questions, specially if technical. I trained models on a collection of royalty free books from the Project Gutemberg , a lot of open-source software, and various datasets available on Huggingface .

I quickly understood that this approach was not feasible; a proper run would have required at least a month of training non-stop. I was amazed by how my models looked alive and magical, but I was also ashamed because they were incredibly wasteful and effectively useless.

Local models

I pivoted to ollama and open-weight models and developed howto , yet another terminal harness that uses a pre-prompt to force the model to answer only with terminal commands. Results were generally unsatisfactory because of the time required to get a response. Models like ornith:9b , mistral:7b or cogito:14b can get the job done sometimes, but they are not fast and reliable enough for general use, specially if you have only 4GB of VRAM.

Going deterministic

Then I remembered about the blockchain craze, when everyone wanted to fit a blockchain somewhere and sell it as the next big thing. I didn't want to waste my time and money like all those people did in the previous hype cycle, so I started building a terminal assistant from scratch with a new set of constraints:

- No embeddings

- No machine-learning

- No LLMs

Dataset format

The first things I needed was a set of conventions to rely on, so I drafted the NDF 0.0 (NPC-Forge Dataset Format which specifies the dataset format of NPC-Forge. The following object contains category, input sentences, textual response, thinking traces, permission gating and tool calls to be executed in a format compatible with VS code.