PokeBench: I gave frontier LLMs the Game Boy screen and 1,000 turns to beat Pokemon Red's first gym

PokeBench tested frontier LLMs by giving them the Game Boy screen, eleven buttons, a walkability grid, and 1,000 turns to beat Pokemon Red's first gym without hints. Out of 14 runs, 4 models succeeded, with GPT-6 Astra winning in 246 turns. Gemini 3.8 Flash achieved the cheapest win at $3.70, while a free local model, Muse Glimmer 30B, only reached Route 1. Full logs and VODs are available at pokebench.tv.

This report uniquely details how different frontier LLMs performed on a standardized gaming benchmark, contrasting their efficiency and cost to beat the first gym in Pokémon Red.