I started a small research project to find product niches: pull keyword export files out of a search portal's ad tools, fold them into one table, and rank what looks promising. Twenty-three spreadsheets, a bit under twenty thousand keywords. The build was the easy half. Learning what the numbers could not tell me took the rest of the evening.
A column that looks like a signal and isn't
The export includes a competition rating, which is exactly the field you reach for when you want to find an underserved niche. Its actual distribution across my data: 17,725 keywords rated high, 2,744 medium, 251 low. Eighty-five per cent in one bucket is not a ranking, it's a constant. The neighbouring field — monthly ad impressions — had 61% of rows sitting at its ceiling value of ten, which is the same problem in a different shape.
Neither field is wrong. They describe competition for ad placement, which is a real thing that happens to not be the thing I was asking about. What I wanted to know was whether the internet is already saturated with content or products for a term. An advertising auction cannot answer that, however confidently it prints a number.
Demand is given to you; supply you have to go and get
That's the whole lesson of the session, and it took a second data source to make it concrete. Search volume — how many people look for a thing — comes free in the export. Supply — how much already exists for that thing — does not exist in the export at all, and has to be collected separately by asking a search API how many documents it holds for each term.
With both halves you get something worth looking at. In a trial run of forty terms, one narrow trade-specific phrase came out with a demand-to-supply ratio of 0.61; a common consumer term in the same category came out at 0.02, because it's searched constantly and already has 460,000 documents behind it. Ranked by search volume alone, the second one wins easily. Ranked by both, it's the worst possible thing to write about.
Three corrections, all of them mine
Nineteen of the twenty-three exports turned out to be truncated at the tool's 1,000-row export cap. Only four were complete. Any conclusion of the form "this seed has about a thousand related terms" was an artefact of the export limit, not a fact about the market.
I reported 1,139 duplicate keywords across seeds, which sounded like an interesting overlap structure. A thousand of them came from two files that were byte-identical exports of the same seed under different filenames. Real cross-seed overlap: 158.
And I asserted that connecting the paid search-ads API would lift the 1,000-row cap. I had not checked. I hadn't read it anywhere; it simply sounded like the kind of thing that would be true. It went into the README as a fact and came back out as "needs measuring" once I noticed.
The part that killed the original goal, for now
The purpose of all this was to find products worth selling — which requires knowing how many sellers already list an item. Partway through, the portal migrated its open APIs behind a new gateway, so my requests needed different authentication headers and a different base path, and endpoints turn out to be enabled per application rather than globally: only two of the eight I wanted were live for my credentials.
More decisively, the shopping search that returns product counts isn't in the new catalogue at all. Not disabled — absent, with the old path returning a plain 404. So the supply side of a product question currently has no source I can reach, and the dashboard's opportunity view had to be rebuilt to answer the narrower question it can actually support: how crowded is the content around this term.
That's a less satisfying result than the one I set out for, and it's the honest one. A tool that scores opportunities using only the half of the data that happens to be available will produce rankings that look complete and mean nothing.