A new agent workflow called ReproBridge enhances machine learning experiment reproducibility by creating standardized, refined artifacts. This approach significantly increases successful reproductions and reduces associated costs.
A study introduces ReproBridge, an agent workflow designed to improve the reproducibility of machine learning research. The system constructs refined research artifacts, which include a cleaned codebase, environment setup scripts, experiment launchers, and a reproduction guide. These artifacts are designed to address the scattered nature of information typically required for reproducing ML experiments. A benchmark, ReproBridge-Bench, comprising 1,029 tasks from 387 experiments across 36 papers, was used to evaluate the system. The benchmark includes five human-verified reference scores for each task to ensure accuracy.
When agents used these refined artifacts instead of raw released code, the mean reproduction success rate increased from 76.4% to 92.6%. This represents an average increase of 16.2 percentage points. The gains varied across different AI models, with increases ranging from 5.5 to 23.2 percentage points. For instance, one model saw a 23.2 percentage point jump in success rate. In addition to improving success rates, the use of refined artifacts also led to a reduction in downstream API spending. The total tokens spent on API calls fell by an average of 19.7%, translating to a cost reduction of approximately 20.6% to 23.7%.
Furthermore, ReproBridge-Bench demonstrated that refined artifacts narrow the spread of success rates between different AI models. The variation in success rates across four evaluated models decreased from 21.6 to 3.9 percentage points. This indicates a more consistent performance across various agents when using the standardized artifacts. An ablation study within the benchmark indicated that removing any component of the refined artifact reduced success rates, with the most significant losses occurring when the code or environment components were removed. This highlights the importance of preserving the full execution knowledge within the reproducible package.
The research acknowledges limitations, including the dataset's scope, which covers 36 papers and 1,029 tasks but not the full spectrum of machine learning research. It also notes that procedures can preserve incorrect assumptions, and the automatic audit might miss certain violations. Future work aims to expand the dataset and refine the audit logic.