nemotron-halloween

**Building a Haunted House with NVIDIA Nemotron Open Models**

Or rather…

**Boo! Now Get Off My Lawn!**

TL;DR: MonsterHouse is an interactive projection I created for Halloween. He listens to trick-or-treaters, answers in the voice of a grumpy old pirate ship and softens when visitors remember their manners. He is built on the NVIDIA Nemotron Voice Agent with NVIDIA Nemotron ASR , Nemotron 3.5 Bolt E4B 4GB , Magpie TTS Zeroshot and Audio2Face-3D , and he runs on a single workstation.

I wanted the kids visiting my house at Halloween to interact in a fun and unique way. The house was going to be in charge and giving out the treats, but more grumpy than scary. Somehow, my solution was to give the garage door a mouth and teach him to complain.

That became MonsterHouse: a grumpy old building with pirate stories, a fiery grin and a suspicious weakness for “please.” NVIDIA Nemotron Voice Agent supplies the conversation foundation. Nemotron Speech does the listening, Bolt E4B is the brain, NVIDIA Magpie TTS Zeroshot is the live voice, and NVIDIA Audio2Face-3D makes the wooden mouth talk.

**How Monster House works**

- Listening: NVIDIA Nemotron 3 ASR turns visitor speech into text. [nemotron-speech-streaming-en-0.6b.q8_0.gguf ]

- Thinking: NVIDIA Nemotron 3.5 Bolt E4B 4GB , an open small language model for conversational agents, writes each reply. A LoRA adapter supplies the personality.

- Speaking: NVIDIA Magpie TTS Zeroshot voices each reply from a short reference sample of the character.

- Animating: NVIDIA Audio2Face-3D turns clean speech into mouth animation in Unreal Engine 5.

- Seeing: NVIDIA Nemotron 3 Nano Omni describes costumes, while YOLO and ByteTrack count and track visitors.

- Running it: NVIDIA Nemotron Voice Agent coordinates everything on one workstation with an NVIDIA RTX PRO 6000 Blackwell Max-Q and an NVIDIA RTX PRO 4000 Blackwell.

**A grin full of splinters**

The mouth came first. I imagined the wooden garage door boards splitting into jagged lips, with the burning flames of hell illuminating the driveway from beyond whenever he opened his cursed mouth. He needed to look ancient, expressive and mildly offended by the existence of visitors.

A few expressions from a house with plenty to say.

Creating a custom animation rig was one of the more labor intensive parts of the build. After bashing it into submission, I could export the facial expression shapes that Audio2Face-3D could later animate. The character SkelMesh was built in Autodesk Maya then exported into Unreal Engine where virtual fire and effects were added behind the door. With no eyes, no eyebrows and no blinking, he’s a surprisingly economical monster.

While all the nonsense of building the character was going on, NVIDIA built the Nemotron Voice Agent. What was supposed to be the hard part of the build now became rather fun, iterative and easy. I had a foundation for listening, conversation and speech right out of the box. The house already had a mouth. Now he could finally find something to complain about.

**Teaching bad manners**

The house needed a past as cantankerous as its personality so I devised a backstory that the house had been built from the timbers of a sunken pirate ship, haunted by its stormy past. The nautical complaints now had a reason. This was the ship, reincarnated as real estate and not entirely pleased about it.

The performance needed a guide, a path to follow from greeting to goodbye while providing treats along the way. First, the house would brush visitors off with remarks like “I’m busy grumbling, go bother the neighbors!” But persistence would wear him down. A “Please” could break through immediately, and “thank you” earned a warm farewell. Once a visitor won him over, I wanted him to stay won over. Nobody wants to negotiate with the same garage door twice.

Here is where the Voice Agent really shines. You can choose any LLM you like. Because I wanted the system to be off-grid and completely self-contained, I had a memory and compute budget.

The character needs more than a prompt that says “be grumpy.” It had to follow a master conversation flow, respond to what visitors actually said, track encounter state, and leave enough compute for the rest of the pipeline.

After testing more than a dozen models, I landed on Bolt E4B to supply the live dialogue. It is an open small language model purpose-built for local conversational agents and interactive characters, which is a fair description of a talking garage door. I use the 4-bit quantized release (the “4GB” in its name) as the default response model, with the BF16 checkpoint available for comparison.