Anyone else feel like their AI feature got expensive?

We shipped an AI feature that looked totally fine in staging, and then real users immediately turned it into a cloud invoice flamethrower.

We had little prompts like summarize this and draft that, maybe a 2k context window if someone got spicy. Then production users showed up with 14 paragraph questions, pasted half their CRM into the box, asked follow-ups with no reset, and our retrieval layer just duplicated the same three snippets because it seemed like one copy of stale policy text was not enough

TBH the dumbest part was how innocent it felt at first. Accuracy looked better with more context, so we stuffed the prompt. Then latency got gross, so we trimmed. Then quality dropped on edge cases, so we added back context. Then someone noticed we were sending nearly identical retrieval chunks plus a giant system prompt plus conversation history every turn.

The budget meeting last week was a nightmare. Nothing like explaining that a user asking a long weird question can cost more than the entire happy path demo flow. And of course the fix is not just reduce tokens. It is chunking, deduping retrieved text, capping history, testing context trimming, and figuring out whether latency or accuracy gets to be the thing everyone complains about this week.

How are you controlling token spend when normal users start doing normal user chaos