I ran two image-generation pipelines head to head — five prompts, six seeds each, 60 images generated in total — and judged the results myself, by hand, through 59 head-to-head rounds (the pairing isn't one round per image; a matchmaking system decides who faces whom). The overall score came out close to a tie: 28 wins to 23, with 8 too close to call. If I'd stopped there, the honest conclusion would have been "no real difference."
Splitting the same results by subject told a completely different story. On one prompt — a fisherman on an overcast day — one pipeline won 12 times out of 12. On another — a mountain ridge shot into backlight — the other pipeline won 14 times out of 15. Two landslides, pointing in opposite directions, cancelling out into what looked like a coin flip. The real finding wasn't "which pipeline is better," it was "which pipeline for which kind of image" — and the average had been quietly erasing that the whole time.
Then I asked an AI to judge, and it had a favorite
To scale past hand-judging every pair, I tried an 8-billion-parameter local vision model as an automatic judge on the same 60 image pairs. Its internal consistency looked fine — it agreed with itself 77% of the time when the same pair was shown twice, and never contradicted itself outright. That looked like a healthy signal.
It wasn't. Across 80 judged pairs, the model picked the same pipeline 63 times, the other pipeline 17 times, and never once called a tie — regardless of subject, including the prompt where a human judge had gone 14-1 the other way. When I had it re-score the exact 20 pairs I'd already judged myself, it matched my calls only half the time. That's chance-level agreement. A judge that always gives the same answer will look perfectly consistent and still be measuring nothing — high agreement with itself is not the same thing as being right. The model's stated reasons were vague, too — "more coherent composition," "better lighting" — and almost never pointed at a concrete flaw like a broken hand or malformed text.
Trying a larger, 27-billion-parameter version of the same model didn't fix it. It was only slightly less biased and, worse, agreed with my own judgments even less often than the smaller model had. Scale wasn't the problem, so scale wasn't the fix.
The question was the bug
The next test explains the first one. I changed the prompt: instead of just picking a winner, the judge had to name a concrete flaw first. Asked that way, it hedged on 75% of the pairs — "both look similar" — even in cases where it had just described a real flaw a sentence earlier. Asking a model to justify a negative before it commits tends to make it retreat instead of decide; the earlier "biased" judge may simply have been reacting to a question that let it avoid ever having to defend an answer.
To check, I brought in a stronger, frontier-scale model and gave it the same small, flaw-first task. On the six pairs it could give a real verdict on, it agreed with my own hand-judged calls on four of six — and its reasoning cited genuine defects, not vague aesthetic language. That reversed my first conclusion: the smaller model wasn't broken, it was answering a version of the question that rewarded picking a side over being right.
Where it landed
The two pipelines split cleanly along two different axes. On technical correctness — hands, anatomy, object logic — one pipeline was consistently more reliable. On overall feel — light, atmosphere, texture — the other pulled ahead, sharply, on specific subjects. Averaging those two axes into one score was the original mistake; keeping them separate is what let the actual pattern show up at all.