OpenAI Audit Reveals Significant Flaws in SWE-Bench Pro Coding Benchmark with 30 Percent of Tasks Estimated Broken

Key Takeaways

• Significant Benchmark Flaws: OpenAI estimates that approximately 30% of the tasks in the SWE-Bench Pro benchmark are broken, affecting the reliability of model evaluations.
• Impact on Safety Decisions: Flawed benchmarks can misrepresent AI capabilities, which directly impacts safety cases and deployment decisions under OpenAI’s Preparedness Framework.
• Rapid Performance Gains Questioned: While frontier models improved their pass rates from 23.3% to 80.3% in eight months, the audit suggests this progress may be obscured by evaluation noise.
• Rigorous Audit Methodology: The investigation involved a multi-stage pipeline including failure trace analysis, investigator-agent passes, and independent reviews by five experienced software engineers.

In-Depth Analysis

The Evolution of Coding Benchmarks: From Verified to Pro

The transition from SWE-bench Verified to SWE-Bench Pro was intended to address fundamental design and contamination issues discovered in earlier evaluations. OpenAI previously found that SWE-bench Verified no longer provided a meaningful signal regarding software development capabilities. In response, the community was encouraged to adopt SWE-Bench Pro, which was specifically engineered to test models on longer horizons and more realistic coding tasks. The goal was to better track agentic coding capabilities by requiring models to implement solutions that pass new feature tests without breaking existing functionality. These tasks are programmatically sourced from the history of feature changes across various public and private repositories.

The Audit Methodology: A Multi-Layered Approach

To separate signal from noise, OpenAI implemented a detailed datapoint analysis pipeline to review SWE-Bench Pro. This pipeline was designed to scrutinize model attempts, task metadata, and failure traces to flag potential evaluation flaws. The process did not rely solely on automated tools; each flagged task underwent a rigorous assessment. This included multiple investigator-agent passes and independent reviews by five experienced software engineers. In cases where disagreements occurred among the reviewers, the tasks were escalated for further review to ensure the highest possible accuracy in identifying broken tasks. This level of scrutiny highlights the complexity of creating reliable benchmarks for agentic AI.

The Gap Between Performance and Accuracy

The data shows a dramatic increase in model performance on the 731-task public split of SWE-Bench Pro, with frontier models jumping from a 23.3% pass rate to 80.3% in just eight months. However, OpenAI’s audit suggests that these numbers may be misleading. If 30% of the tasks are broken, the "signal" regarding a model's true software development capability becomes difficult to distinguish from the "noise" of flawed tasks. When evaluations have flaws that affect results, they can give a false understanding of capabilities, which in turn misrepresents safety cases and affects research priorities. This finding underscores the importance of continuous auditing of the benchmarks used to track AI progress.

Industry Impact

Redefining AI Safety and Deployment Standards

The revelation that a major coding benchmark contains 30% broken tasks has significant implications for the AI industry. Accurate measurement of model capabilities is not just a matter of academic interest; it is a critical component of sound deployment and safety decisions. For organizations like OpenAI, these benchmarks inform decisions made under frameworks like the Preparedness Framework. If the industry relies on flawed metrics, it risks deploying models based on a misunderstood sense of their safety and functional limits. This audit may prompt a broader industry-wide shift toward more rigorous, human-verified benchmarking processes.

The Future of Agentic Coding Evaluations

As AI models move toward more agentic behavior—performing complex, multi-step tasks over longer horizons—the benchmarks used to measure them must become equally sophisticated. The issues found in SWE-Bench Pro suggest that programmatic sourcing of tasks from repository histories, while efficient, may introduce significant noise. The industry may need to move toward evaluation sets that are more resistant to contamination and design flaws. OpenAI’s use of investigator-agents and human experts to audit these benchmarks sets a new precedent for how AI evaluation datasets should be maintained and verified to ensure they provide a meaningful signal for future research.

Frequently Asked Questions

Question: Why did OpenAI conduct an audit of SWE-Bench Pro?

OpenAI conducted the audit because accurately measuring model capabilities is vital for making sound deployment and safety decisions. They found that flaws in evaluations can lead to a false understanding of AI progress and misrepresent safety cases under their Preparedness Framework.

Question: What were the specific findings regarding the tasks in SWE-Bench Pro?

The audit estimated that approximately 30% of the tasks in SWE-Bench Pro are broken. This follows a previous investigation into SWE-bench Verified, which was found to have fundamental design and contamination issues that prevented it from providing a meaningful signal on software development capabilities.

Question: How did the pass rates of frontier models change during the audit period?

On the 731-task public split of SWE-Bench Pro, frontier models showed a significant improvement in pass rates, rising from 23.3% to 80.3% over a period of eight months. However, the audit suggests these results must be viewed in the context of the identified task issues.