A new benchmark, AI4AI-Bench, has been introduced to evaluate the ability of large language model (LLM) agents to design and improve training algorithms, a core component of recursive self-improvement (RSI) in artificial intelligence. The benchmark, developed by researchers at Navers Lab and Einsia.AI Tsinghua University, addresses a gap in existing evaluation methods that often credit performance gains to data collection or hyperparameter adjustments rather than fundamental algorithmic changes.
Recursive self-improvement posits that an AI system could enhance the process that creates subsequent AI systems, leading to a continuous cycle of improvement. This process hinges on an AI's capacity to modify its own training algorithms, such as objective functions or update rules, to achieve a more efficient compute-capability exchange rate. AI4AI-Bench aims to isolate and measure this specific capability.
The benchmark consists of 10 distinct tasks, each derived from a frozen research repository representing a different family of training algorithms. In each task, an LLM agent is allocated four hours on a B300 GPU to modify the existing training algorithm's source code. The agent's submission is the altered code, not a numerical output or a trained model.
Following the four-hour modification period, the submitted code is executed from scratch for up to 12 hours under a fixed computational budget. A predetermined evaluator, hidden from the agent, then scores the resulting model against the repository's original algorithm. The scoring system normalizes performance across the 10 incommensurable metrics, mapping results onto a scale where 0 represents an uninformative model, 0.1 signifies the original algorithm's performance, and 1.0 denotes the task optimum.
Initial experiments with 29 configurations of six different systems across all 10 tasks yielded a mean score of 0.166. The best-performing system achieved a score of 0.250, indicating that even the most capable agents closed less than one-fifth of the gap between the baseline algorithm and the optimal solution.
The analysis of submissions revealed that most agents did not alter how the model learns. However, the minority of submissions that did modify the learning mechanism achieved an average score of 0.226, compared to 0.126 for those that did not. Increased reasoning effort by the agents correlated with a higher propensity to modify the learning mechanism, raising this minority from 8% to 64% of submissions and improving the mean score from 0.094 to 0.196.
The researchers have made the task suite, evaluators, and all scored submissions publicly available. This transparency allows for reproducibility of measurements as AI systems evolve.
Existing AI benchmarks frequently focus on areas such as multi-turn dialogue performance, problem-solving across various domains, or resolving real-world software issues. Benchmarks like MT-Bench and BIG-Bench assess language models' reasoning and extrapolation abilities, while SWE-bench evaluates their capacity to generate code patches for GitHub issues. However, these benchmarks do not specifically isolate an agent's ability to design or improve the underlying training algorithms. The introduction of AI4AI-Bench aims to provide a targeted evaluation for this critical aspect of recursive self-improvement.