OpenAI released a blog post publicly challenging the industry authority evaluation benchmark SWE-Bench Pro, stating that about 30% of the 731 public test tasks have evaluation flaws. SWE-Bench Pro was launched by Scale AI and is specifically designed to evaluate the programming capabilities of large language models and AI agents. Due to its high similarity to real-world enterprise development and extremely high anti-cheating standards, it has now become the industry authority benchmark in the field of AI software engineering.

OpenAI

OpenAI pointed out a key signal in the blog post: the pass rate of cutting-edge models on this benchmark increased from 23.3% to 80.3% in just 8 months. This "progress" speed is too abnormal, and OpenAI believes that this benchmark can no longer effectively evaluate the true software development capabilities of models. It is likely due to systematic issues within the evaluation itself, rather than a real leap in model capabilities.

Two review paths cross-validated, nearly 30% of tasks "unqualified"

To verify this judgment, OpenAI initiated two parallel review paths. The data point analysis process identified 200 failed tasks, accounting for 27.4% of the total 731 public tasks; at the same time, the manual annotation activity identified 249 failed tasks, accounting for 34.1%. Based on the cross-validation of these two paths, OpenAI estimates that about 30% of the tasks in SWE-Bench Pro have defects, involving four types of issues: overly strict testing, insufficient prompts, narrow testing scope, and misleading prompts.

OpenAI also disclosed a typical case: one question required adding one space at the beginning of a line when converting content to Markdown, but the hidden test required two spaces. This means that even if the model wrote code according to the problem statement, it would still be marked wrong. This kind of "hidden requirements inconsistent with explicit instructions" directly leads to an incorrect assessment of the model's true capabilities, and explains why the pass rate has seen an unreasonable surge.

Withdraw the adoption recommendation, call for rebuilding the AI evaluation system

Based on this analysis, OpenAI officially withdrew its previous recommendation to adopt SWE-Bench Pro. OpenAI believes that in the future, new benchmarks should be specifically designed by experienced software developers for AI evaluation, rather than simply using the testing logic intended for human developers. When the industry "benchmark" itself may have nearly 30% of defects, the credibility of the entire AI evaluation system faces scrutiny. Returning from score-chasing competitions to real engineering capability assessments may be the next essential step in AI software engineering evaluation.