ScarfBench: Benchmarking AI Agents for Enterprise Java Framework Migration

ScarfBench is introduced as an open benchmark to evaluate AI agents on enterprise Java framework migration, a complex task beyond simple code translation. It assesses whether migrated applications build, deploy, and preserve behavior across Spring, Jakarta EE, and Quarkus. Current agents show low behavioral success rates, often overestimating their completion. Migration is iterative, with agents frequently revisiting configuration, and struggles extend to environmental and tooling issues, highlighting that the biggest challenge isn't just code transformation.