OpenAI's Frontier Evals: The Contamination of SWE-Bench Verified and What's Next

The Stalemate of SWE-Bench Verified

• 💡 SWE-Bench Verified, once a key benchmark for coding progress, is now considered saturated and contaminated.
• 🎯 Recent analysis shows that many failures on SWE-Bench Verified are due to unfair tests or narrow specifications, not true model inability.
• ⚠️ OpenAI will stop reporting SWE-Bench Verified results due to these limitations.

The Evolution and Challenges of Benchmarking

• 🚀 SWE-Bench Verified was initially a significant cleanup of the original SWE-Bench, involving extensive human review by nearly 100 software engineers.
• 🔍 However, models began recalling repository-specific details or task identifiers, indicating contamination.
• 📈 The benchmark's high performance ceiling means further small improvements are less meaningful and may reflect an agent's ability to guess specific naming conventions rather than true coding capability.

Transitioning to SWE-Bench Pro

• ✅ OpenAI will now focus on SWE-Bench Pro from Scale, which is harder, more diverse (more repos and languages), and includes longer tasks.
• 📊 SWE-Bench Pro shows substantially less evidence of contamination based on OpenAI's analysis with a contamination auditor agent.
• 🚀 The goal is to move towards benchmarks that better reflect real-world coding challenges and model capabilities.

Future of Coding Benchmarks

• 🧩 Ideal coding benchmarks should measure beyond simple pass/fail tests, including open-ended design decisions and code quality/maintainability.
• 🛠️ Evaluating longer-horizon tasks, real-world product building, and subjective qualities like