When Modernization Breaks Working Code: A Study of LLMs on Legacy PHP

A study on modernizing legacy PHP code with LLMs found that simply asking an LLM to "modernize this" often breaks working code. Researchers evaluated five LLMs on 100 WordPress 4.0 PHP files, aiming to update them to PHP 8.3 while preserving behavior. The LLMs broke 32-51 files, repairing at most 10, compared to Rector which broke 7 and repaired 6. Much of this damage was unnecessary, as only 37 findings required changes for PHP 8.3 compatibility.