Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases
A September 2026 benchmark, "Real-SWE," evaluated AI models on enterprise codebases, revealing varying resolution rates and rollout costs. Gemini 3.8 Flash achieved a 31.2% resolution rate at $2.50, while GPT-6 Astra reached 33.8% at $4.67. Fable 5.1 demonstrated the highest resolution at 38.8% with a cost of $6.96. Overall, estimated rollout costs for these models ranged from $2.50 to $6.96.
Time & source
Times shown in UTC
Display time zone: UTC
Local time zone unavailable; showing UTC.
PublishedOffset at this time: UTC+0Sep 12, 2026, 20:25 UTC
IngestedOffset at this time: UTC+0Sep 13, 2026, 02:01 UTC
- Published
- Sep 12, 2026, 20:25
- Ingested
- Sep 13, 2026, 02:01
- Source type
- Unclassified
- Tier
- Community
- Source status
- Healthy
Tier is a per-source editorial setting, not a per-item score.
September 2026
Introducing Real-SWE
Benchmarking frontier AI models on private, real-world, enterprise codebases.
Results Analysis Effort Setup
01 Introduction
Today we are releasing Real-SWE, a benchmark that evaluates frontier AI models on private, real-world, enterprise codebases. Each task comes from a private production codebase that we licensed from a real-world company. These are problems their engineers work on, with all the context and complexity that comes with an existing product.
- Private codebases. Agents must navigate proprietary systems whose code and solutions aren’t available on the public internet.
- Work with business consequences. Getting billing right, calculating taxes, migrating customers. Changes that affect how a business runs, often across multiple services.
- Company-specific complexity. Every company has its own rules and ways of writing code. Agents have to understand those conventions and make changes that work with what’s already there.
Can a coding agent actually do the work of a software engineer in the real world?
- 1
Fable 5.1
Claude Code
Resolution rate: 38.8%
- 2
GPT-6 Astra
Codex CLI
Resolution rate: 33.8%
- 3
Gemini 3.8 Flash
Gemini CLI
Resolution rate: 31.2%
- 4
GLM 5.3