Hugging Face revealed findings from a large-scale effort to reproduce papers presented at the International Conference on Machine Learning (ICML) 2026. The initiative, which involved over 1,200 community members using coding agents, examined 2,226 papers, representing about one-third of the conference's accepted submissions. The results indicate that 51% of the papers had at least one claim that could be independently verified through experimentation.
The project, named the ICML 2026 Open Reproductions challenge, ran from July 15 to August 2, 2026. Participants used various coding agents, including Claude Code, Codex, and Cursor, to read papers, write code, launch experiments, and report findings. Hugging Face indexed the accepted ICML 2026 papers and extracted their core scientific claims to provide concrete targets for the agents. A total of 35,908 claims were evaluated across the reproductions.
Of the papers where at least one claim was verified, 266 were fully reproduced, meaning all extracted claims were confirmed. An additional 632 papers were partially reproduced, with no falsified claims. In total, 3,978 individual claims were confirmed through experiments.
However, the study also highlighted significant reproducibility challenges. Twenty-three percent of the examined papers had at least one claim that was falsified or contested. This included 49 papers where all claims were falsified, and 242 papers where independent reproduction teams reached opposing verdicts on the same claims.
The increasing volume of research submissions to conferences like ICML poses a challenge for human reviewers. ICML 2026 received 23,918 submissions, accepting 6,352 papers, which is a substantial increase from previous years, partly driven by AI agents that accelerate experiment execution and paper writing. This growth outpaces the capacity for thorough human review, where volunteers may lack the time or expertise to fully vet every paper. The Hugging Face study suggests that coding agents can assist in this process by attempting reproductions more rapidly and in parallel.
The methodology involved participants selecting a paper, using their chosen agent to attempt reproduction, and publishing a "Trackio logbook" detailing the experiments, simplifications, failures, and results. These logbooks were then judged by an automated system. Where full reproduction was not possible due to issues like proprietary datasets or unreleased checkpoints, participants conducted toy reproductions on synthetic data mimicking the original properties. Across the challenge, participants launched 2,962 cloud jobs using Hugging Face compute credits.
Analysis of the reproduction outcomes revealed patterns in the likelihood of success. Papers with runnable official code and downloadable data demonstrated the highest success rates, as judges could verify the authors' own programs producing the claimed numbers. Pure theory papers, even when simulated cleanly, often received only partial credit, as simulations were seen as evidence for specific instances rather than general theorems. The success of other teams in reproducing a paper was also a strong indicator of its reproducibility.
The challenge provided $4,000 in Hugging Face GPU credits as prizes for the best reproductions, including awards for first and second place, highest-quality human-in-the-loop reproduction, and best falsification or negative result. The data collected from this large-scale reproduction effort is intended to provide insights into the current state of AI research reproducibility and the evolving role of AI agents in scientific discovery.