When local AI can't solve a coding problem on the first try, what actually helps? Most of what I tried did nothing — two things took my solve rate from 42% to 75%.

The problem. When a model fails at a coding problem, the usual move is to re-prompt it and hope for better results. That’s a good intuition, but more of a feel than a rule. I wanted to measure the payback for repeating prompts. I’d completed Advent of Code in 2024 and 2025, which gave me a useful set of interesting problems with objectively correct answers to test model-generated code against — so I gave a local model multiple attempts at them, varying one thing at a time to see what matters most.

Most of what I varied moved nothing — not raising the temperature, not rewording the error feedback. Only two changes helped, and both gave the model information it didn’t already have.

What I did. I tested on the band where the model is unreliable — problems it sometimes solves and sometimes doesn’t. Easy ones it always gets, impossible ones it never gets, so the uncertain middle is the only place retrying can make a difference. The model was qwen3.8:27b, running locally under Ollama.

approach solved 1 attempt, allowed to fix its own errors 42% (10 of 24) 3 attempts, no feedback 58% (7 of 12) 3 attempts, each allowed to fix its own errors 75% (9 of 12) More attempts help. Feedback helps. Together they help more than either does alone.

The size of my tests is admittedly small:

- Four problems, three runs each — Advent of Code 2024, days 13 and 15, both parts. The single-attempt row has 24 draws behind it rather than 12 because I pooled it with an earlier run on identical settings.

- I picked those four before running the experiment, from a separate scoring run, so I wasn’t choosing problems that would flatter the result.

- Small enough that I repeated the whole thing on a year I’d never measured. The pattern held: on AoC 2025, 2 of 8 became 6 of 6.

- Across the project, 43 solutions have been verified against real answers, none wrong.

Then the part I didn’t expect. One problem has now failed 13 out of 13 times across every configuration I’ve tried: one attempt, three attempts, with feedback, without, at two different temperatures. The model makes the same wrong assumption every single time.

Where the failure is systematic rather than random, extra attempts just re-roll the same die.

I tried the obvious fixes for that: raise the temperature to get more variety in the attempts, and reword the error feedback to be more pointed. Neither moved anything. All told, five of my predictions about how this would behave turned out to be wrong — which became the most useful thing I got out of the project.

What the failures had in common: they changed how the same request was phrased or sampled, without giving the model anything it didn’t already have. A higher temperature shuffles word choices inside the same wrong approach. Sharper wording restates an error the model has already seen.

The two that worked bring something in from outside. A fresh attempt is an independent draw rather than a perturbation of the last one, so it can land on a different approach entirely. A repair step hands the model an actual traceback — a fact produced by running the code, not by the model guessing about its own code.

Interventions that add no new information don’t help, however well worded they are.

So the rule I’d offer anyone building a retry loop: before adding an intervention, ask what the model actually learns that it didn’t already know. If the answer is “nothing”, it probably won’t help.

How this compares. The underlying ideas aren’t mine — repeated sampling and self-repair both have published research behind them.

The difference is in what varies. Benchmarks like HumanEval, SWE-bench and Aider’s polyglot suite vary the model and hold the strategy fixed. This holds the model fixed and varies the strategy. The heavier ones also need API spend or containers to reproduce; this runs on local hardware.

Where this lands. The sampling result agrees with the repeated-sampling work: more independent draws raise coverage at a model’s frontier. The repair result agrees with the self-repair paper, which found repair limited by the quality of the feedback rather than by the retrying — execution output is just a cheaper source of that quality than asking a model to critique its own code. ( Large Language Monkeys , Brown et al., arXiv 2407.21787; Is Self-Repair a Silver Bullet for Code Generation? , Olausson et al., arXiv 2306.09896, ICLR 2024.)

Where my results push back is on diversity. A two-model ensemble I predicted would union to 6/8 came back 5/8 — the same solve set as the better model alone, at about 2.2x the wall-clock. The apparent complementarity was sampling noise: the solve I was counting on from the second model had been a lucky draw, not a reliable strength.

If you’re building a voting orchestration, that’s what I’d check first: measure each member alone on your targets before pooling them, or you’ll pay for a second model that adds nothing.

For scale: the headline model is 17 GB and runs on a consumer Mac, and the whole body of work behind these numbers is 57 experiment runs, roughly 121 GPU-hours and 16.3M tokens, at zero API spend.

Fair warnings: it’s one model on one machine, and AoC puzzles are public and predate these models, so treat “how capable is it” as soft. The comparisons above run the same model against itself on the same problems, so contamination can’t explain the differences between them.

Repo is MIT and public: https://github.com/geniusworks/problem-solver . Everything runs on local models through Ollama, so you can point it at your own hardware without an API bill. Full statistics are in RESULTS.md , including the parts that didn’t work.