GPT-6 Astra vs GPT-5.6 Sol: benchmark on 50 real PRs, looking for feedback on the methodology

A benchmark comparing GPT-6 Astra and GPT-5.6 Sol was conducted on 50 real PRs from projects like Cal, Sentry, Discourse, Keycloak, and Grafana. GPT-5.6 Sol identified 107 confirmed bugs compared to GPT-6 Astra's 91, though Astra demonstrated higher precision and lower latency. All findings were independently verified. The evaluators are seeking feedback on their methodology before benchmarking Fable vs Opus next week.

This report uniquely offers a direct comparison of GPT-6 Astra and GPT-5.6 Sol's bug detection capabilities on 50 real-world pull requests, unlike typical benchmarks.