When a local model fixes a broken script, does a second model — arguing with the first rather than agreeing — actually make the answer better? We built a small test harness to measure it on real home-server bugs, and the result was clear enough to change how we think about small-model accuracy.
Everything here runs on our own hardware, so the models are the ones that fit on a pair of consumer graphics cards, not a frontier model behind an API. The worry with those is accuracy: a fix that looks plausible, and even passes a quick check, can still be wrong. The idea we wanted to test is an old one — have two models of different families each propose a fix, then take turns finding flaws in each other’s until they agree — and the question was simply whether it helps enough to be worth the extra time.
The suite is 44 cases, each one a real bug from this house: a Home Assistant automation that fires when it should not, a shell script that falls over under set -u, a compose file with smart quotes in it, a backup that quietly wipes itself when a drive is missing. Every case ships a broken version, a visible check whose output the model is allowed to see, and a hidden test that the model never sees and that actually decides whether the fix is right.
The gap between those two is the whole point. A change that makes the visible check pass but fails the hidden test is a false pass — the dangerous kind, because it looks finished. A few of the cases are traps where the right answer is to make no change at all and say the fault is elsewhere.
It runs on the compute server: two graphics cards, with one language-model family resident on each so nothing has to reload between turns. In round zero, both models propose a fix on their own and the visible check runs on each. Then they alternate — each reads both proposals and what the check said about them, names concrete flaws in the other’s, answers the objections to its own, and then keeps its position, adopts the other’s, revises, or abstains. This goes up to five rounds or until they land on the same change. The prompts are written to push hard against agreeableness, because the obvious failure is two models being polite and agreeing on nonsense.
Each model answered all 44 cases on its own, same prompt, same settings. “Right” means the hidden test passed, or a correct refusal on a trap case; false passes are fixes that satisfied the visible check but failed the hidden one.
| Model | Size | Right | False passes |
|---|---|---|---|
| Gemma 4 31B | 31B dense | 33 | 5 |
| Qwen3.8 27B | 27B dense | 32 | 5 |
| Qwen3.8-Flash-Next 125B | 125B mixture-of-experts | 31 | 6 |
| Qwen3.8 27B, smaller quant | 27B dense, 3-bit | 30 | 6 |
| Nemotron 3.5 Super VL 120B | 120B mixture-of-experts | 30 | 6 |
| Nemotron 3.5 Bolt 74B | 74B mixture-of-experts | 28 | 5 |
| Liquid LFM2.5 24B | 24B mixture-of-experts | 18 | 7 |
The striking part: the big models did not pull ahead. A 120B, a 74B and a 125B all landed at or below two mid-sized models that fit on the cards at once — and the bigger they were, the slower.
The pair we took forward were Qwen3.8 27B and Gemma 4 31B — the two strongest small models in the table — one on each card, chosen from different families so they do not share the same blind spots. Debating, they reached 38 right with 5 false passes: past every single model above, including the ones three and four times their size, and at least five cases better than either of them managed alone.
The gain came from a clear place: about six cases a run that neither model solved on its own but the discussion got right, while the debate almost never talked a model out of a correct answer it already had. We also tried letting the two share their full private reasoning rather than just their proposals, and it made no measurable difference. The cost is roughly six model calls and five minutes of GPU time per case.
The honest weak spot is those false passes, which barely moved. When both models land on a fix that passes the shallow check, the debate has nothing concrete to argue about and tends to settle. So the next step is to make the discussion argue from evidence rather than words: agreement on a change that fails the check never counts, a model cannot simply give in without something to back it up, and — the interesting part — each model may propose a small test of its own that the harness actually runs on both fixes, so a claim about the other’s fix gets checked instead of merely asserted. That version is running as this page goes up.