My AI-written tests kept passing broken code, so I started testing the tests

I’m not really a programmer. I’ve mostly been making and patching stuff by telling LLMs what I want and then checking whether it actually works.

One thing kept pissing me off though. I’d ask it to fix something, it would write a test, the test would pass, and then later I’d find out the test wasn’t actually checking what I thought it was checking.

So at some point I started doing a pretty dumb simple thing. After something passed, I’d intentionally break the exact behavior the test was supposed to protect and run the test again.

If it still passed, then obviously the test was bullshit.

I ran into this with a plugin that had a paged archive. The archive test was green, so I changed the code so only part of the archive would get copied. Test still passed.

Then I made the test stricter, and managed to screw it up in the opposite direction. The new test rejected the correct code too, because I had made a bad assumption about how the archive pages would be laid out.

So I threw that one away as well.

Eventually I got a test where the normal version passed and the deliberately broken version failed for the reason I actually cared about. While doing that I also found a real bug where a failed write could leave part of the stored data changed.

After this happened enough times I kind of stopped treating “tests passed” as the end of the story.

I started doing the same thing to the stuff that decides whether the tests passed. Basically asking things like, if part of the checking process never ran, could it still say everything was fine? If I gave it old or altered results, would it notice? If it claimed it could rebuild the final result from the original, did it actually do that?

It has gradually turned into this whole verification procedure I use when I’m patching things with LLMs.

I later found out mutation testing is obviously related to what I was doing, but most of what I’ve read about mutation testing is about breaking the program to see whether the tests notice. What I ended up doing also treats the tests and the rest of the checking process as things that can be wrong.

I’ve tried it on a few different plugins/projects now and it has found enough weird false-passes that I’m starting to wonder whether this is actually useful outside my own workflow or whether I just invented an absurdly overcomplicated way of checking AI-written code.

Has anyone here done something similar, or is there an existing methodology/tool I should be looking at?