I had called the largest of three local vision models the best one. It was a 32-billion-parameter build, and it had handled my standard set of three images well. Two newer 27-billion-parameter builds handled them well too. I had treated a test that all three passed as evidence for a ranking it could not actually produce.
Keep the images, change the questions
Rather than run the same easy prompts again, I asked harder questions about the same three images: count small objects, read exact lettering, identify tools, and say which way a beam was travelling. I checked the answers against the images by hand and ran the models in isolation under the same request settings.
The older, larger build missed the beam direction. It also invented a line of text that was not in the image and left out tools that were there. The newer builds did better on those details, with one of the 27-billion-parameter models giving the cleanest answers across this small test. On this battery, my earlier “largest is best” conclusion did not survive.
What the test can say
This was three images and a set of deliberately difficult questions, not a universal ranking of vision models. It was enough to show that the old three-image check had saturated: once everyone passes, repeating it cannot distinguish the candidates. The next comparison needs questions that expose their weak points, with answers fixed before reading the model output.
Speed needed its own column. A model that generates tokens quickly can still feel slow when it writes much more than its competitors. I separated decoding speed from total response time so a long answer would not masquerade as a slow engine. The practical change was small: stop letting one comfortable test and one large parameter count make a decision that neither had earned.