Why decontamination reports can't fix benchmark contamination, and what an evaluator has to do instead [D]

OpenAI stopped reporting SWE-bench Verified in February, recommending other labs do the same, as frontier models could reproduce human-written fixes or problem details for some tasks. Progress had slowed, and the benchmark's creator retired it. The author, who built a similar system, acknowledges limitations like the benchmark's quality, the potential for repeated submissions to compromise hidden test sets, and the risk of leaked labels, prioritizing closing the gap related to repeated submissions.

This post uniquely details the challenges of benchmark contamination, unlike typical reports that focus on decontamination, and highlights the specific limitations of current evaluation methods.