Debates surrounding the accuracy and transparency of AI benchmarks have recently taken center stage, with a public dispute emerging between OpenAI and xAI over the reported performance of the Grok 3 AI model. The controversy began when an OpenAI employee accused xAI of presenting misleading benchmark results for Grok 3, sparking a heated discussion within the AI community.
In response to these accusations, Igor Babushkin, one of the co-founders of xAI, defended the company’s position, claiming that their benchmark results were accurate. However, the truth behind these claims is not as clear-cut as it may seem.
In a blog post on xAI’s website, a graph was shared showcasing Grok 3’s performance on AIME 2025, a set of challenging math problems derived from a recent mathematics exam. While some experts have raised concerns about the validity of AIME as an AI benchmark, it is commonly used to evaluate a model’s mathematical capabilities.
The graph presented by xAI depicted two variations of Grok 3, Grok 3 Reasoning Beta, and Grok 3 mini Reasoning, outperforming OpenAI’s best-performing model, o3-mini-high, on AIME 2025. However, OpenAI employees quickly pointed out that xAI’s graph failed to include o3-mini-high’s score at “cons@64,” a crucial metric that significantly impacts the final benchmark results.
To provide context, “cons@64” refers to “consensus@64,” a method that gives a model 64 attempts to answer each problem in a benchmark, with the most frequently generated answers being considered final. Excluding this metric from the graph could create a misleading impression that one model surpasses another when, in reality, this may not be the case.
Upon closer examination, it was revealed that the initial scores for Grok 3 Reasoning Beta and Grok 3 mini Reasoning on AIME 2025 fell below o3-mini-high’s score at “@1.” Furthermore, Grok 3 Reasoning Beta lagged slightly behind OpenAI’s o1 model set to “medium” computing. Despite these findings, xAI continues to market Grok 3 as the “world’s smartest AI.”
In response to the ongoing debate, a more impartial observer created a revised graph that accurately depicted the performance of each model at cons@64. This new graph shed light on the discrepancies between xAI’s claims and the actual benchmark results, showcasing a more comprehensive view of the models’ capabilities.
AI researcher Nathan Lambert emphasized the importance of considering the computational and monetary costs associated with achieving each model’s best score. This critical metric underscores the limitations and strengths of AI models, highlighting the need for a more holistic evaluation approach.
As the discussion surrounding AI benchmarks continues to evolve, it is essential to maintain transparency and accuracy in reporting performance metrics. By considering a range of factors, including computational costs and benchmark methodologies, a clearer picture of AI capabilities can be established, guiding future advancements in the field.
Kyle Wiggers, a senior reporter at TechCrunch specializing in artificial intelligence, offered valuable insights into the complexities of AI benchmarking. With a keen interest in emerging technologies, Wiggers brings a unique perspective to the ongoing debate, shedding light on the nuances of AI performance evaluation. His expertise and dedication to uncovering the truth behind benchmark claims provide a valuable resource for those navigating the rapidly changing landscape of AI research.
As the AI community grapples with the implications of misleading benchmark results, the need for transparency and accountability grows ever more crucial. By engaging in open dialogue and promoting data-driven evaluation practices, stakeholders can work together to ensure the integrity of AI benchmarking processes, fostering innovation and advancement in the field.
