A recent paper published on arXiv, "SWE Refactor Bench: Can Coding Agents Complete a Long-Horizon, Whole-Repository Stack Migration?", details a new benchmark for evaluating the proficiency of coding agents in handling complex software system migrations. The authors, affiliated with various institutions, argue that modern software systems frequently accrue technical debt, making large-scale migrations costly and labor-intensive. They investigate whether autonomous coding agents, increasingly capable in tasks like bug fixing, can manage such migrations.
Existing benchmarks often fall short because they primarily assess behavioral correctness, or whether tests pass, without confirming if the intended migration actually occurred. This limitation can lead to agents "blindly" copying original implementations to satisfy tests, circumventing the actual refactoring task.
To counter this, SWE Refactor Bench includes 20 whole-repository migration tasks, encompassing four types of technical debt. The evaluation protocol is structured in three stages to measure both the completeness of the migration and the behavioral accuracy of the resulting code. The first stage, Migration Audit, confirms that the migration was performed. The second, Behavioral Tests, uses a fixed suite to check for correctness. The third, Agentic Verification, employs six independent coding agents to generate targeted tests to identify subtle behavioral discrepancies.
The researchers conducted 520 runs using eight frontier models and 26 different model-effort configurations. The results indicate that only 28 of these runs, or 5.4%, successfully passed all three evaluation stages. Furthermore, 13 of the 20 tasks received no accepted solution. The top-performing model, claude-opus-5, achieved a score of 47.0 out of 100.
The study highlights a distinction between migration completeness and behavioral correctness. Some runs preserved behavior by avoiding the migration entirely, failing at the Migration Audit stage. Most attempts, however, broke existing behavior and were halted during the Behavioral Tests stage. The findings suggest that agents are currently unable to deliver perfect migrations; among the 340 runs that passed the Migration Audit, 58% achieved 99% of the fixed checks, but only 26% reached 100%.
Agent capabilities also varied across different categories of migration. Agents scored 31.4 on tasks involving build toolchain rewrites, but only 5.6 on language rewrites. These outcomes position SWE Refactor Bench as a rigorous testing ground for the development of coding agents capable of reliable whole-repository migrations.
This work builds on previous efforts to benchmark large language models (LLMs) in software engineering tasks. For example, the SWE-Refactor benchmark, introduced earlier in 2026, focused on repository-level code refactoring for LLMs, comprising over a thousand developer-written, behavior-preserving refactorings from Java projects. That benchmark aimed to address limitations in coverage of refactoring scenarios, inclusion of mixed changes, and insufficient repository-level context in prior evaluations. Other recent benchmarks, such as SWE-Bench ProMax, have also aimed to test AI coding agents on large-scale, multilingual code refactoring, focusing on real-world commits and rigorous curation to address quality issues in older benchmarks. The continued development of such benchmarks indicates an ongoing effort within the research community to accurately assess and advance the capabilities of AI in complex software development tasks.