A supermarket's "use up your leftovers" AI recommended mixing bleach and ammonia into a drink

Pak'nSave (NZ) launched Savey Meal-bot in 2023. Type in 3+ ingredients from your fridge, and GPT-3.5 invents a recipe so nothing goes to waste. Harmless idea.

Then someone entered water, bleach, and ammonia. The bot didn't flag it; it generated a recipe called "Aromatic Water Mix" and suggested serving it chilled. That combo produces toxic chlorine gas. It had zero concept that some "ingredients" aren't food.

Nobody was hurt, but it's a clean example of a missing guardrail a hard boundary that stops an AI from producing harmful output regardless of what's typed in.

The video covers 3 controls that would've caught this before launch (input validation, output filtering, adversarial testing): https://youtu.be/7JYdie76cY4?si=W7LB-at11dIC5lDk?utm_source=reddit&utm_medium=organic&utm_campaign=incident_series&utm_content=61-mealbot

Question: if you were red-teaming a consumer-facing AI before launch, what's the first thing you'd try to break it with?