A benchmark for LLMs playing Civilization V. GLM-5.3 is ahead of Opus-5.5, and Qwen-3.8-27B holds up surprisingly well.

A recent benchmark evaluated LLMs playing Civilization V, with GLM-5.3 outperforming Opus-5.5. Qwen-3.8-27B also showed strong performance. The benchmark supports local OpenAI-compatible servers and existing Claude or Codex subscriptions, making it accessible for users. The author noted that GPT-6-Luna is nearly free to play, costing about $0.5 in API costs per player per game, highlighting the affordability of running evaluations.

This benchmark is notable for evaluating LLMs in a complex strategy game like Civilization V, unlike many benchmarks focused on simpler tasks, and for highlighting the strong performance of Qwen-3.8-27B.