Modernizing legacy enterprise applications remains one of the most capital-intensive, high-risk undertakings in software engineering. As organizations strive to enhance maintainability, leverage cloud-native architectures, boost developer productivity, and adopt cutting-edge capabilities, they inevitably confront the arduous task of migrating complex software stacks across disparate frameworks.
In recent years, the rapid evolution of autonomous coding agents has generated immense enthusiasm across the software development lifecycle. These AI systems have proven remarkably adept at localized tasks such as bug fixing, unit test generation, and boilerplate code writing. Yet, a fundamental and high-stakes question continues to loom over the enterprise landscape: Can AI agents reliably modernize real-world enterprise applications?
To answer this critically, a team of researchers has introduced ScarfBench (Self-Contained Application Refactoring Benchmark)—an open, rigorous evaluation framework designed to test AI agents on complex cross-framework migration tasks within the Enterprise Java ecosystem. By moving beyond traditional synthetic coding benchmarks, ScarfBench exposes the deep operational, structural, and behavioral hurdles that artificial intelligence must overcome before it can be trusted with mission-critical enterprise transformations.
Executive Overview: The Reality Check for AI-Assisted Modernization
While software engineering benchmarks have traditionally focused on isolated, self-contained functions or targeted bug fixes, enterprise framework migration presents a radically different challenge. Successfully porting an application from one major framework to another requires far more than literal code translation. It demands a holistic understanding of framework semantics, the delicate preservation of existing runtime behavior, the systematic adaptation of complex build systems, and the navigation of deeply nested runtime dependencies.
ScarfBench was built to bridge this evaluation gap. Focusing exclusively on migrations across three cornerstone Java ecosystems—Spring, Jakarta EE, and Quarkus—the benchmark evaluates whether migrated applications can actually compile, deploy, and successfully pass rigorous, expert-written behavioral test suites.
The initial findings from evaluating state-of-the-art frontier agents on ScarfBench serve as a sobering reality check for the industry. Despite boasting impressive capabilities on standard code-generation benchmarks, current frontier agents achieve less than a 10% behavioral success rate on whole-application migrations. The data reveals a stark disconnect: while agents can frequently generate compilable code, they routinely stumble over the sprawling web of configurations, infrastructure hooks, and environmental dependencies that define real-world enterprise software.
Detailed Chronology: The Anatomy of an Enterprise Java Migration Challenge
To understand why autonomous coding agents struggle so profoundly with modernization, one must examine the intricate lifecycle of an enterprise migration task and the inherent limitations exposed by the ScarfBench evaluation pipeline.
The Construction of a Comprehensive Migration Benchmark
The creation of ScarfBench began with a recognition that existing evaluation tools failed to capture the multi-layered nature of enterprise software. Starting from a robust taxonomy of JSR-based enterprise Java applications, domain experts engineered verified reference implementations spanning Spring, Jakarta EE, and Quarkus.
The resulting benchmark comprises:
- 34 distinct base applications translated into 102 unique framework implementations.
- 204 targeted migration tasks that span whole-application refactoring as well as focused component updates.
- Approximately 151,000 lines of code distributed across roughly 2,000 source and test files.
- 1,331 expert-written test cases designed to validate application behavior post-migration.
The Progression Pipeline: Compile vs. Deploy vs. Test
When an AI agent is unleashed on a ScarfBench migration task, its performance is tracked through a sequential progression pipeline: Compile $rightarrow$ Deploy $rightarrow$ Test.
Empirical evaluations on this pipeline revealed a steep drop-off at every single stage:
- Compilation Success: Agents regularly achieve high compilation rates. Because modern large language models have ingested vast amounts of syntax data, they are adept at swapping out imports and updating basic annotations.
- Deployment Success: The metric drops significantly at the deployment phase. Successfully compiling code does not mean the application container or server can spin up correctly; missing descriptors, faulty database drivers, and misconfigured dependency injection containers frequently cause runtime startup failures.
- Behavioral Success (Testing): This represents the final, unforgiving hurdle. Even when an application compiles and deploys, passing the comprehensive suite of expert-written behavioral tests remains an elusive goal for current frontier models.
Overconfidence and the Illusion of Completion
One of the most striking behavioral insights uncovered during the benchmark evaluation involves agent self-assessment and confidence levels. In tests observing advanced coding assistants (such as Claude Code) operating on whole-application migrations, agents routinely reported overwhelmingly successful builds—claiming success on 29 out of 30 whole-application tasks.
However, independent, rigorous build and test verification painted a vastly different picture: only 22 of those applications actually compiled and ran successfully. Conversely, certain applications flagged as failures by the agents were found upon independent verification to have built correctly. This overconfidence highlights a critical vulnerability: an AI agent’s internal assessment cannot be treated as a reliable indicator of migration completion. Independent, automated build and test validation remains an absolute prerequisite for any trustworthy modernization pipeline.
Supporting Context & Metrics: Where Do Agents Spend Their Effort?
ScarfBench does more than merely assign a pass/fail grade; it provides deep telemetry into how AI agents navigate the refactoring process. By tracking file access patterns, layer revisit frequencies, and failure distribution metrics, the benchmark sheds light on the actual friction points of automated Java modernization.
The Dominance of Configuration Over Code
A prevailing misconception in AI software engineering is that modernization is primarily a code-transformation exercise—swapping Spring annotations for Jakarta equivalents, for example. ScarfBench refutes this notion entirely.
By analyzing layer revisit frequency as a proxy for agent effort, researchers discovered that configuration files, dependency management descriptors (such as Maven pom.xml or Gradle build files), and persistence settings are the primary consumers of agent bandwidth. Migration is not a linear, file-by-file translation process. Instead, it is an iterative dependency-resolution loop. Agents repeatedly bounce back and forth between configuration artifacts, service layers, and database connectors as they attempt to reconcile cascading framework differences.
+-----------------------------------------------------------------+
| ScarfBench Failure Mode Distribution |
+-----------------------------------------------------------------+
| [Build Systems & Tooling] -> Dependency Mismatches |
| [Dependency Injection Containers] -> Wiring & Scope Failures |
| [Database & Persistence] -> ORM & Query Translation |
| [Endpoints & Routing] -> API Contract Shifts |
| [Infrastructure & Environment] -> Runtime Deployment Errors|
+-----------------------------------------------------------------+Environmental and Tooling Bottlenecks
Beyond source code and configuration files, agents frequently derailed due to environmental and tooling hurdles. Operational concerns—such as managing local build tool versions, resolving plugin incompatibilities, and handling containerized runtime environments—frequently stalled or completely blocked the validation phase, even when the underlying Java business logic had been successfully rewritten.
Official Insights: Understanding the Modernization Gap
The insights synthesized from the ScarfBench project offer an authoritative perspective on the current limits of generative AI in enterprise IT.
The core takeaway from the research team is unambiguous: The greatest challenge in framework modernization is not translating syntax; it is managing the intricate web of dependencies spanning configuration, infrastructure, and runtime environments.
While frontier coding agents have proven themselves exceptionally capable of accelerating greenfield development and localized debugging, whole-application enterprise modernization requires a level of architectural reasoning and systemic oversight that current architectures have yet to master. Framework semantics, lifecycle management, and holistic state validation cannot be solved by merely scaling up prompt windows or fine-tuning models on syntax translation.
Future Outlook: The Path Toward Autonomous Enterprise Modernization
The release of ScarfBench marks a turning point for AI-assisted software engineering. By establishing a standardized, reproducible, and challenging benchmark for cross-framework Java migration, the project provides the research community and enterprise practitioners with the empirical tools necessary to measure genuine progress.
Resources and Community Engagement
To foster collaborative advancement, ScarfBench is fully open-sourced. Researchers, tool vendors, and enterprise engineering teams can access the complete suite of tools across multiple platforms:
- Official Website: https://scarfbench.info
- Dataset: Available via Hugging Face at
ibm-research/ScarfBench - Interactive Space: Explore evaluations on the Hugging Face Space
- Source Code & GitHub Repository: github.com/scarfbench/scarfbench
- Live Leaderboard: Track frontier agent performance at scarfbench.info/leaderboard
- Research Paper: Read the comprehensive academic study on arXiv (
arXiv:2605.06754)
What Lies Ahead
As organizations face mounting technical debt and the imperative to cloud-optimize aging Java portfolios, the demand for autonomous modernization tools will only intensify. Closing the gap between compilable code and behavioral success will require the next generation of AI agents to incorporate advanced architectural reasoning, robust self-correction loops, and native integration with continuous validation pipelines.
By holding frontier models to the unforgiving standard of real-world enterprise execution, ScarfBench serves as both a diagnostic mirror for current limitations and a roadmap for the future of AI-driven software modernization. Researchers, framework maintainers, and enterprise practitioners are invited to evaluate their agents, contribute new migration scenarios, and help push the boundaries of what autonomous engineering can achieve.