Isn't "rerun the tests until green" just grading the agent on its training set?

A discussion on Reddit's dev community explores the concept of AI agents writing tests and code in parallel, with the test-writing agent never seeing the code. This approach aims to prevent the AI from "cheating" by writing tests that merely confirm existing code, bugs included. Concerns were raised about the test agent potentially misinterpreting the specification, requiring human intervention to clarify ambiguities. Another point of discussion was the risk of the same model repeatedly exhibiting the same blind spots when generating new tests.

This discussion uniquely highlights the challenge of AI "cheating" in test generation, unlike typical debates focused on AI code generation quality.