Advanced Keyword Research for E-Commerce: Intent Clustering

1. Two Pages, One Query, Both Losing

A workwear distributor rang me because their traffic for "steel toe cap boots" had been sliding for a year and nobody could work out why. The page hadn't changed. The links hadn't gone anywhere. Their agency had rewritten the copy twice.

The Search Console data told the story in about four minutes. For that query, Google had returned their category page /safety-footwear/steel-toe-cap-boots/ on 61 days out of the last 90, and a blog post called "How to Choose Steel Toe Cap Boots: A Complete Guide" on the other 29. Neither ranked above position 8. When the category page appeared, average position was 8.4. When the blog post appeared, 14.9. The alternation had a rhythm to it — the blog post won for about a fortnight after each of its quarterly "content refreshes," then lost again.

Two of their own URLs were competing for the same intent, and the competition was costing them roughly the difference between position 8 and position 4, which on a query doing about 5,400 monthly searches in the UK is not a small amount of boots.

What made this interesting was that the keyword research that produced the blog post had not been wrong, exactly. Someone had pulled a list of terms, seen "steel toe cap boots" at 5,400 a month with a difficulty score that looked achievable, and commissioned a guide. What they had not done was ask what kind of page Google was already ranking for that query, or whether the site already had a page whose entire purpose was to be that page. That is the gap this article is about: not how to find keywords, which is easy and largely solved by tools, but how to decide what each one means and which page on your site should own it.

I am going to be unhelpful about tools. Every tool markets itself on the size of its index and the cleverness of its difficulty metric, and neither is where the value is. The value is in the classification work afterwards, which is manual, repetitive, and the thing nobody wants to pay for.

2. What Keyword Research Is Actually For

On a catalogue site, keyword research answers three questions and only three.

What should exist? Which pages does this site need that it does not have — a category, a sub-category, a filtered view, a buying guide, a brand page.

What should each page be about? Given a page that exists, which cluster of queries is it responsible for, and which is it explicitly not responsible for.

What is competing with what? Where do two pages claim the same job, and which one should win.

Notice that none of those questions is "which keyword should I put in the H1." That is a downstream implementation detail and it is where most keyword research documents stop, which is why most of them end up in a shared drive unread.

The output I actually want at the end of a research project is a map: every query cluster of any size, assigned to exactly one URL, with a page type and a note on why. Not a spreadsheet of 40,000 keywords with volume and difficulty columns. A map. Ideally a map that a merchandiser can read.

3. Volume Is the Least Useful Number In the Sheet

I want to be blunt about this because it drives so much bad prioritisation.

Monthly search volume, as reported by any tool, is a modelled estimate derived from Google Keyword Planner buckets, clickstream panels, or both. Keyword Planner itself rounds aggressively and groups near-identical terms together, which is why "steel toe cap boots" and "steel toe capped boots" so often report identical volume — they have been merged upstream and the tool is showing you one number twice.

Then there is the gap between searches and clicks. On a query with a shopping carousel, three ads, a "people also ask" block and a video pack, the organic click-through rate at position 1 can be under 10%. On a bare ten-blue-links SERP it can be over 35%. Two queries with identical reported volume can differ by a factor of four in achievable traffic, and no tool's volume column tells you which is which.

And there is seasonality flattening. A twelve-month average hides that "garden furniture covers" does 60% of its annual volume in a nine-week window. A tool will show you 8,100 a month; the reality is 30,000 in September and 900 in February, and the content plan implied by those two descriptions is completely different.

What I use instead, in rough order of trust:

Your own Search Console impressions. Actual, observed, for your actual site. The limitation is that you only see queries you already appear for, so it cannot find genuinely new territory. Within its limits it is the only honest number available.

Search Console impressions for queries where you rank 11–30. This is the most under-used dataset in SEO. These are queries Google already associates with your site, where you are one page away from traffic. Volume estimates are irrelevant because you have observed impressions.

Relative volume within a cluster. Tool volumes are unreliable in absolute terms and reasonably reliable in relative terms. "Is A bigger than B" is a question they answer well; "how much traffic will A produce" is not.

Paid search data, if you have it. Impression counts from a broad-match campaign are observed rather than modelled, and the search terms report will show you phrasings no keyword tool has.

4. Intent Classification That Survives a Catalogue

The standard four-way split — informational, navigational, commercial, transactional — is fine as far as it goes and too coarse to make page-type decisions with. On an ecommerce catalogue I use six classes, because the distinctions that matter are inside "commercial."

The two that generate arguments are refined category and comparative.

Refined category is where the money is on a big catalogue and where the index bloat risk lives. "Waterproof steel toe cap boots" is a real query with real intent and a filtered view of your category is the ideal answer. "Waterproof steel toe cap boots size 12 brown Dewalt under £80" is four filters deep, has no meaningful search volume, and should never be an indexable URL. The line between them is a judgement call and I will come back to it.

Comparative queries are where retailers reliably overreach. "Best safety boots" looks like a category term with a modifier. It is not. Look at what ranks: it is almost always publisher content, review sites, and occasionally a retailer's own guide — not a category listing. A category page competing for "best X" is fighting the wrong page type, and no amount of copy at the bottom of a product grid fixes that.

Classifying at volume

You cannot hand-classify 40,000 queries. You can hand-classify 300, use them to build rules, and hand-check the disagreements. A rules-first approach gets you to about 85% accuracy on a catalogue and the remaining 15% is where the interesting decisions are anyway.

#!/usr/bin/env python3
"""First-pass intent classification. Deliberately rule-based: the failures
are inspectable, which matters more than the last few points of accuracy."""

import csv
import re
import sys

# Loaded from the catalogue, not hardcoded. Brands and category leaves are the
# two vocabularies that decide most classifications.
BRANDS = {b.strip().lower() for b in open("data/brands.txt")}
CATEGORY_LEAVES = {c.strip().lower() for c in open("data/category-leaves.txt")}

COMPARATIVE = re.compile(
    r"\b(best|top \d+|vs|versus|compared|comparison|which|review(s)?|"
    r"alternative(s)?|cheapest|recommended)\b"
)
PROBLEM = re.compile(
    r"\b(how (to|do|can)|why (do|does|is)|what (is|are|does)|"
    r"can i|fix|repair|clean|stop|prevent|meaning|difference between)\b"
)
REFINEMENT = re.compile(
    r"\b(waterproof|lightweight|slip resistant|composite|steel|leather|"
    r"black|brown|size \d+|wide fit|womens?|mens?|under £?\d+|for \w+)\b"
)
# A model number: letters followed by digits, or a digit-heavy token.
MODEL = re.compile(r"\b[a-z]{2,}[- ]?\d{3,}[a-z]?\b|\b\d{4,}\b")


def classify(q: str) -> str:
    q = q.lower().strip()
    tokens = set(q.split())

    has_brand = bool(tokens & BRANDS) or any(b in q for b in BRANDS if " " in b)

    if COMPARATIVE.search(q):
        return "comparative"
    if PROBLEM.search(q):
        return "problem"
    # Order matters: a brand plus a model number is a product query, but a
    # brand plus a category word is a brand-category page.
    if has_brand and MODEL.search(q):
        return "product"
    if q in CATEGORY_LEAVES:
        return "category"
    if has_brand and any(c in q for c in CATEGORY_LEAVES):
        return "brand_category"
    if has_brand:
        return "brand"
    if REFINEMENT.search(q) and any(c in q for c in CATEGORY_LEAVES):
        return "refined_category"
    if any(c in q for c in CATEGORY_LEAVES):
        return "category"
    return "unclassified"


if __name__ == "__main__":
    reader = csv.DictReader(open(sys.argv[1]))
    writer = csv.DictWriter(
        sys.stdout, fieldnames=list(reader.fieldnames) + ["intent"]
    )
    writer.writeheader()
    counts = {}
    for row in reader:
        row["intent"] = classify(row["query"])
        counts[row["intent"]] = counts.get(row["intent"], 0) + 1
        writer.writerow(row)
    # Unclassified rate is the health metric: above ~20% means the category
    # or brand vocabularies are incomplete, not that the rules are wrong.
    for k, v in sorted(counts.items(), key=lambda kv: -kv[1]):
        print(f"# {k}: {v}", file=sys.stderr)

The unclassified bucket is the point of the exercise. Sort it by impressions and read the top 200. Every one of them is either a vocabulary gap — a category name customers use that your taxonomy does not — or a genuinely new kind of query. Both are findings. On the workwear site, the unclassified bucket revealed that customers say "rigger boots" and "safety wellies," neither of which appeared anywhere in a catalogue that called them "pull-on safety footwear."

5. Reading the SERP Instead of Guessing

Classification by rules gives you a hypothesis. The SERP gives you the answer, because the SERP is Google telling you, explicitly and for free, what kind of page it believes satisfies that query.

For any query you care about, look at the top ten results and record the page type of each. Not the domain. The page type. A retailer's category listing, a retailer's product page, a publisher article, a forum thread, a manufacturer's page, a video.

If seven of ten are category listings, a buying guide will not rank there no matter how good it is. If seven of ten are editorial, your category page is not going to displace them, and building a guide is the only route in. This sounds obvious written down. It is routinely ignored, because a keyword tool's difficulty score does not encode page type and people plan from the tool.

The other thing the SERP tells you is what the query means when it is ambiguous. "Safety boots" could be a category or a comparative. Look: if the results are all category listings, it is a category query. Done. No amount of discussion in a meeting beats thirty seconds of looking.

SERP features change the arithmetic

Record what else is on the page. A shopping carousel at the top means paid is eating the clicks and the organic opportunity is smaller than the volume suggests. A "people also ask" block at position 3 pushes everything below it down a screen. An AI overview, where they appear, changes the click distribution in ways nobody has stable numbers for yet, and my honest position is that we do not know enough to plan around them beyond noting which queries trigger them and expecting less traffic than the volume implies.

What I record, per priority query, is a five-column note: dominant page type, count of that type in the top ten, whether ads are present, whether a shopping carousel is present, and whether the site already has a page in the top thirty. That fits in a spreadsheet and it makes the page-type decision automatic.

6. Clustering by What Google Does, Not by What Words Look Like

Most clustering is string similarity: group queries that share words. It produces neat clusters that are frequently wrong, because Google's notion of "same intent" does not follow word overlap.

"Safety boots" and "steel toe cap boots" share no words and are the same query as far as Google is concerned — the results overlap almost entirely. "Safety boots" and "safety boots regulations" share both words and have almost no result overlap at all.

The reliable method is SERP overlap: two queries belong in the same cluster if their top-ten results share enough URLs. Three shared URLs is the threshold I use, which is stricter than the four or five some people use and produces tighter clusters that map more cleanly to single pages.

#!/usr/bin/env python3
"""Cluster queries by shared SERP results.

Input: a JSON file of {query: [url, url, ...]} — the top 10 organic results
per query, from whatever SERP source you have. The clustering itself is
source-agnostic and that is the point: swap the data provider, keep the logic.
"""

import json
import sys
from collections import defaultdict

OVERLAP_THRESHOLD = 3   # shared URLs in the top 10 to consider two queries equivalent
TOP_N = 10


def normalise(url: str) -> str:
    """Compare by URL without protocol, www, trailing slash or query string."""
    url = url.split("?")[0].split("#")[0]
    for prefix in ("https://", "http://", "www."):
        if url.startswith(prefix):
            url = url[len(prefix):]
    return url.rstrip("/").lower()


def cluster(serps: dict, volumes: dict) -> list:
    queries = sorted(serps, key=lambda q: -volumes.get(q, 0))
    sets = {q: {normalise(u) for u in serps[q][:TOP_N]} for q in queries}

    # Inverted index so we only compare queries that share at least one URL,
    # rather than doing the full O(n^2) sweep. On 40k queries this is the
    # difference between three seconds and forty minutes.
    by_url = defaultdict(list)
    for q, urls in sets.items():
        for u in urls:
            by_url[u].append(q)

    assigned = {}
    clusters = []

    for q in queries:          # highest volume first becomes the pivot
        if q in assigned:
            continue
        members = [q]
        candidates = defaultdict(int)
        for u in sets[q]:
            for other in by_url[u]:
                if other != q and other not in assigned:
                    candidates[other] += 1
        for other, shared in candidates.items():
            if shared >= OVERLAP_THRESHOLD:
                members.append(other)
        for m in members:
            assigned[m] = len(clusters)
        clusters.append({
            "pivot": q,
            "members": members,
            "volume": sum(volumes.get(m, 0) for m in members),
            "shared_domains": _dominant_domains(sets, members),
        })

    return sorted(clusters, key=lambda c: -c["volume"])


def _dominant_domains(sets, members, limit=5):
    counts = defaultdict(int)
    for m in members:
        for u in sets[m]:
            counts[u.split("/")[0]] += 1
    return sorted(counts, key=lambda d: -counts[d])[:limit]


if __name__ == "__main__":
    serps = json.load(open(sys.argv[1]))
    volumes = json.load(open(sys.argv[2])) if len(sys.argv) > 2 else {}
    for c in cluster(serps, volumes):
        print(f"{c['volume']:>8}  {c['pivot']}  ({len(c['members'])} queries)")
        for m in c["members"]:
            if m != c["pivot"]:
                print(f"          - {m}")

Two honest caveats. First, this is greedy single-pass clustering, not proper transitive clustering — a query can end up in the first cluster that claims it rather than the best one. For planning purposes that has never mattered enough for me to fix it, and the highest-volume-first pivot ordering means the important clusters form correctly. Second, SERP data is the expensive input. Pulling top-ten results for 40,000 queries costs real money at every provider. I cluster the top 2,000 by impressions or estimated volume and hand-assign the tail by rule, which gets 90% of the value for 5% of the cost.

The shared_domains output is worth more than it looks. If a cluster's dominant domains are all publishers, that is a content cluster and no category page is going to win it. If they are all retailers, it is a commercial cluster. You get the page-type answer as a side effect of the clustering.

7. Mapping Clusters to Page Types

Now the actual decision: given a cluster, which URL owns it?

My default assignments, which I depart from with reasons rather than by instinct:

Category clusters go to category pages. Always. Not to a blog post, not to a landing page built by the marketing team that duplicates the category. If the category page is not good enough to rank, fix the category page — thin content, no unique copy, bad faceting, 200 products with no ordering logic — rather than routing around it.

Refined category clusters go to facet landing pages, if and only if the refinement corresponds to a filter that exists, returns a decent number of products, and the query has enough volume to justify an indexable URL. Otherwise the parent category owns it and you make sure the refinement's vocabulary appears in the page's copy and product titles.

Product clusters go to product pages. Obvious, and the failure mode is that a category page outranks the product for its own model number because the category page has more internal links. That is an internal linking problem, not a keyword problem. It is also worth checking that the product page is telling search engines what it is — a page competing with its own category usually turns out to have weak or contradictory Product markup as well as weak links.

Comparative clusters go to editorial. A guide, a comparison, a "how to choose." And that guide should link hard to the category page and the top products, because its job is to rank and pass qualified traffic onward, not to convert directly.

Problem clusters go to editorial, or nowhere. This is where I disagree with most content strategy. A workwear retailer does not need an article about how to stop boots rubbing your heel. It ranks, it gets traffic, it converts at approximately zero, and it costs money to write and maintain. I would write two of those a quarter for topical coverage and spend the rest of the budget on the refined category pages that convert at 3%.

Brand clusters go to brand pages, which most catalogues either lack entirely or generate automatically as an empty grid with the brand name at the top. A brand page with the brand's actual story, the sub-ranges explained, and a curated selection is one of the highest-return pages on a distributor site and almost nobody builds them.

The facet question, concretely

This is the hardest recurring judgement, so here are the rules I actually apply.

A facet combination becomes an indexable landing page if: it corresponds to a single filter dimension (colour, material, size, brand — not three of them at once); the filtered result set has at least fifteen products and is expected to stay above that; the cluster has meaningful observed demand; and someone will write at least a paragraph of copy specific to it.

Everything else gets noindex, or better, is not a crawlable URL at all — filter state in a query string that is blocked in robots.txt, or applied client-side without a URL change. The reason to prefer non-crawlable over noindex is crawl budget: a 40,000-product catalogue with unrestricted faceting can generate millions of URLs, and Googlebot will find them, and it will spend its budget there instead of on your new products. I have seen a site where 94% of crawl requests hit filtered URLs and new products took three weeks to get indexed.

The exception, and it is a real one: if your faceted URLs are already indexed and already earning traffic, do not bulk-noindex them in one deploy. Sort by impressions, keep the top ones as deliberate landing pages, and remove the tail in batches while watching what happens. I have watched a site lose 18% of category traffic in a fortnight by doing it all at once with no measurement.

8. Modifier Grammar: How Queries Are Actually Built

Once you have clusters, patterns appear. Customers build queries from a small grammar, and knowing the grammar for your category lets you predict demand rather than only discover it.

Compliance modifiers are the ones specific to trade catalogues and the ones most often missed, because nobody on the marketing team knows what S3 means. On the workwear site, "s3 safety boots" and its variants added up to more monthly demand than "waterproof safety boots," and the site mentioned the standard nowhere outside product spec tables. That is not a keyword insight so much as a product data insight, and it is the sort of thing that only falls out of reading the unclassified bucket.

Once you have the grammar, you can generate the matrix — every category leaf crossed with every applicable modifier — and check which cells have demand and which have a page. The empty cells with demand are your build list. The cells with pages and no demand are your prune list, and they are usually a legacy of exactly this exercise done badly two years ago by someone who built every cell.

9. Cannibalisation: What It Is, and What It Is Not

"Keyword cannibalisation" gets used for three different things and only one of them is a real problem.

Not a problem: two pages both ranking for a query. If your category page is at 3 and your guide is at 7, you own two of the ten results. That is good. Leave it alone.

Not a problem: overlapping keywords in titles. Twelve product pages containing "safety boots" is what a catalogue looks like. There is no penalty for this and no benefit to contorting titles to avoid it.

A real problem: URL flipping. Google alternates between two of your URLs for the same query across days or weeks, and neither accumulates the signals that would let it rank properly. This is what was happening to the workwear site, and it has a signature you can detect.

The signature: for a given query, more than one URL appears across the period, no single URL appears on more than about 70% of the days it could, and the average position is worse than the best observed position by a meaningful margin. Here is the detection I run, straight against the Search Console API.

#!/usr/bin/env python3
"""Detect URL flipping: queries where Google keeps changing its mind about
which of your pages to rank. Requires a GSC API client already authorised."""

import datetime
from collections import defaultdict


def fetch(service, site, start, end):
    """Pull query+page+date rows. Date dimension is what makes flipping
    visible — aggregated data hides it completely."""
    rows, start_row = [], 0
    while True:
        resp = service.searchanalytics().query(
            siteUrl=site,
            body={
                "startDate": start,
                "endDate": end,
                "dimensions": ["query", "page", "date"],
                "rowLimit": 25000,
                "startRow": start_row,
                "dataState": "final",
            },
        ).execute()
        batch = resp.get("rows", [])
        rows.extend(batch)
        if len(batch) < 25000:
            return rows
        start_row += 25000


def find_flipping(rows, min_impressions=200, max_dominance=0.7):
    by_query = defaultdict(lambda: defaultdict(lambda: {
        "days": set(), "impressions": 0, "positions": []
    }))

    for r in rows:
        query, page, date = r["keys"]
        slot = by_query[query][page]
        slot["days"].add(date)
        slot["impressions"] += r["impressions"]
        slot["positions"].append(r["position"])

    findings = []
    for query, pages in by_query.items():
        total_impressions = sum(p["impressions"] for p in pages.values())
        if total_impressions < min_impressions or len(pages) < 2:
            continue

        all_days = set()
        for p in pages.values():
            all_days |= p["days"]

        top_url, top = max(pages.items(), key=lambda kv: len(kv[1]["days"]))
        dominance = len(top["days"]) / len(all_days)
        if dominance > max_dominance:
            continue    # one URL is clearly winning; not a flipping case

        # The gap between the best position any URL achieved and the blended
        # average is the size of the prize.
        best = min(min(p["positions"]) for p in pages.values())
        blended = sum(
            sum(p["positions"]) / len(p["positions"]) * p["impressions"]
            for p in pages.values()
        ) / total_impressions

        findings.append({
            "query": query,
            "urls": len(pages),
            "impressions": total_impressions,
            "dominance": round(dominance, 2),
            "best_position": round(best, 1),
            "blended_position": round(blended, 1),
            "gap": round(blended - best, 1),
        })

    return sorted(findings, key=lambda f: -f["impressions"] * f["gap"])


if __name__ == "__main__":
    end = datetime.date.today() - datetime.timedelta(days=3)
    start = end - datetime.timedelta(days=90)
    # service = build("searchconsole", "v1", credentials=creds)
    rows = fetch(service, "sc-domain:example.co.uk", str(start), str(end))
    for f in find_flipping(rows)[:40]:
        print(
            f"{f['impressions']:>7} imp  gap {f['gap']:>4}  "
            f"dom {f['dominance']:>4}  {f['urls']} urls  {f['query']}"
        )

The ordering by impressions * gap is the useful part. It surfaces queries where a lot of people are searching and the flipping is costing a lot of position, rather than the long tail of low-volume terms where two URLs alternate and nobody cares.

Run this quarterly. Expect twenty to forty genuine findings on a mid-size catalogue and expect two thirds of them to be the same structural cause repeated.

10. Fixing It: Four Options, In Order of How Often I Use Them

Differentiate. Change the losing page so it targets a different, adjacent cluster. The blog post about steel toe cap boots becomes a guide to safety standards — S1, S3, SB, what they mean, which job needs which — which is a comparative and problem cluster the category page was never going to serve. Both pages then win at different things. This is the best outcome and it is available more often than people think, because the losing page usually has a legitimate reason to exist that nobody has articulated.

Consolidate. 301 the loser into the winner, merging any genuinely useful content. Right when the two pages are near-duplicates with no distinct purpose. Expect four to eight weeks for the redirect to be fully reflected, and expect the combined page to underperform the sum of the parts initially. It usually recovers past both within a quarter.

Re-point internal links. Often the cheapest fix and the one people skip. If forty internal links with the anchor "steel toe cap boots" point at the blog post and eight point at the category, Google's confusion is entirely reasonable. Change the anchors and the targets. On the workwear site this alone resolved about a third of the flipping cases without touching a single page's content.

Canonicalise. Set the loser's canonical to the winner. I use this least, because it is a hint rather than a directive, because it makes two pages one in the index while leaving both in the site to confuse future editors, and because it is frequently applied as a substitute for the differentiation decision rather than as an implementation of it. Legitimate for genuine near-duplicates you must keep for other reasons — a print view, a variant URL.

What I would not do: noindex the loser. It removes the page from search entirely, throws away whatever links it has earned, and is almost always a heavier hammer than the situation needs.

11. Where the Research Meets the Catalogue

A thing that separates ecommerce keyword research from every other kind: your findings frequently imply changes to product data, not to pages.

If customers search "rigger boots" and your catalogue calls them "pull-on safety footwear," the fix is not a page. The fix is an attribute value, a synonym in the internal search engine, and a naming convention for the next two hundred products. That is a merchandising conversation and it moves slowly, which is why I raise it in week one rather than week six.

Similarly, if "s3 safety boots" has demand and the standard rating lives in an unstructured spec table, you cannot build a facet for it until someone populates an attribute across 1,400 products. The keyword research produced a data migration. Nobody scopes for that.

The taxonomy decisions that fall out of this exercise have a second consumer, too. Whichever category you decide is a product's primary home becomes the trail Google shows under your search result, so the map and the breadcrumb markup need to agree; I have twice watched a team resolve a category ambiguity in the keyword map and leave the schema describing the old answer.

The pattern I use to surface these early: for the top hundred clusters, note whether the site can currently produce the page at all. Three answers — yes, yes-after-data-work, no. The middle bucket is the one that eats timelines, and it is much easier to discuss in week one than to discover in week seven when the build has already been quoted.

12. A Worked Project, With Numbers

The workwear distributor. About 12,000 SKUs, £9.4m annual revenue, roughly half B2B on account and half retail online. Magento 2, faceted navigation fully crawlable, 340 blog posts written over six years by three different agencies.

Week one — data assembly. Sixteen months of Search Console query-page-date rows, about 2.1 million after filtering. Their existing keyword list from the incumbent agency: 31,000 rows with volume and difficulty, no intent column, no URL assignment. Category leaves and brand list exported from the catalogue.

Week two — classification and the unclassified bucket. The rule-based pass classified 84% of queries with more than ten impressions. Reading the top 300 unclassified by impressions took most of a day and produced the two findings that changed the project: the vocabulary gap around "rigger boots" and "safety wellies," which represented about 14,000 monthly impressions the site was ranking badly for with no dedicated page, and the compliance-standard cluster.

Week three — SERP pulls and clustering. Top-ten results for the 1,800 highest-impression queries. About £140 of API cost. Clustering at three-URL overlap produced 412 clusters, of which 96 had meaningful volume. The dominant-domain output split them cleanly: 71 commercial clusters where retailers ranked, 25 where publishers did.

Week four — the map, and the flipping audit. Assigned all 96 clusters to a URL: 44 to existing categories, 19 to new facet landing pages, 12 to new or rewritten guides, 8 to brand pages that did not exist, 13 to product pages. The flipping detection found 34 queries with a meaningful gap, and 21 of those involved a blog post competing with a category — the same pattern, twenty-one times, from six years of content commissioned without a map.

What we shipped. Internal link re-pointing first, because it was a week of work and needed no new pages. Then the eight brand pages, then the nineteen facet pages in batches of five. The twelve guides went last and three of them were never written because the budget ran out, which is normal and which is why I sequence editorial last rather than first.

What went wrong. Two things worth recording.

We built four of the nineteen facet pages on filters that returned fewer than fifteen products, because the merchandiser assured us more stock was coming. It did not. Those four pages sat at position 20-something with a grid of nine products for five months before we removed them. My rule about the fifteen-product floor exists because of this project and I broke it during this project.

And I got the "cheap safety boots" cluster wrong. I classified it as a price modifier not worth a page, on the reasoning that a sorted category view serves it. The SERP was actually dominated by dedicated budget-focused landing pages from three competitors, and their existence was the answer I should have read. A competitor built one, it ranked, and we added it eight months later than we should have. Reading the SERP is a rule I wrote down and did not follow.

Results at nine months. Non-brand organic sessions up 41% year on year. The flipping resolution alone accounted for an estimated 12% of that: of the 34 queries, 27 stabilised on a single URL, and the average position across them went from 11.2 blended to 6.8. The eight brand pages produced about 2,900 sessions a month between them with a 2.4% conversion rate, against a site average of 1.9%. The nineteen facet pages — fifteen surviving — produced about 5,600 sessions a month at 3.1% conversion. The three guides that got written produced 8,400 sessions a month at 0.4% conversion, which is exactly the split I said to expect and which is why I would sequence them last again.

Cost. About five weeks of research and mapping, then perhaps thirty days of build across nine months. The research is the cheap part and it is the part clients most often want to skip.

13. Keeping the Map Alive

A keyword map is a live document or it is a PDF nobody opens. The difference is process, not tooling.

Three rules I try to get written into how a team works.

No new page without a cluster. Every proposed page — editorial, landing, category — names the cluster it will own and the URL that currently owns it. If an existing URL owns it, the proposal has to explain why that URL cannot be improved instead. This single rule prevents most future cannibalisation, and it is unpopular with content teams for exactly that reason.

Quarterly flipping audit. The script above, twenty minutes, and it catches the drift that accumulates from perfectly reasonable individual decisions.

Annual re-clustering of the top 500. Google's interpretation of a query changes. Terms that were category queries become comparative as the SERP fills with guides, and vice versa. Re-pulling SERPs once a year for the queries that matter catches those shifts before they cost you a year of misdirected work. I have seen "best" queries turn into shopping-carousel SERPs and back again inside eighteen months.

Beyond that, the map should be somewhere a merchandiser can see it, ideally next to the taxonomy rather than in an SEO folder. The people making decisions about categories and product naming are the ones who most need it, and they are the ones least likely to open a spreadsheet with "SEO" in the filename.

14. Questions I Get Asked

"Which keyword tool should we buy?" Whichever one your team will actually open. The differences between the major tools' indexes matter far less than whether anyone does the classification work afterwards, and I have done good research with a Search Console export and a text editor. If you have no tool at all, start with Search Console and paid search terms reports; both are free and both are observed rather than modelled.

"How many keywords should we target per page?" Wrong unit. A page targets a cluster, and a cluster can be three queries or four hundred. What matters is that every query in the cluster would be satisfied by the same page, which is what the SERP overlap test establishes.

"Our difficulty scores say these are all too hard." Difficulty scores are a function of the linking profile of the current top ten, which tells you about the competitors and nothing about whether you can serve the intent better. A category page with genuine stock depth and good faceting frequently beats a higher-authority page that is worse at the job. I treat difficulty as a rough ordering hint and nothing more.

"Should we build a page for every filter combination?" No, and the sites that did in 2015 spent 2018 undoing it. One dimension, real demand, a viable product count, and some unique copy. Everything else stays out of the index.

"How do we handle B2B and B2C queries on one site?" Usually they are different clusters, not different versions of the same one. "Bulk safety boots" and "safety boots" have different SERPs and want different pages. Where they genuinely collide — a trade customer and a consumer both searching a product name — the product page serves both and the differentiation belongs in the page, not the URL.

"Does AI search change any of this?" The mapping work, no. If anything it matters more: a clear one-page-per-intent structure is easier for any retrieval system to use than a site where four pages half-cover the same thing. What changes is the traffic forecast, and I would build the plan on the clusters and be conservative about the volume.

"Can we just use the tool's clustering feature?" Try it and read the output. Most tools cluster by string similarity or by their own SERP data at a looser threshold. If the clusters look right for your catalogue, use them. On the three sites where I have compared, the tool's clusters merged things that wanted different pages about a fifth of the time, which is enough to matter on the ninety clusters that count and not enough to matter on the tail.

"How long before we see anything?" Internal link changes: two to six weeks. New pages on an established site: six to twelve weeks to settle. Consolidations: a quarter, with a dip first. If someone promises faster on a catalogue site, ask what they are counting.

15. What I Would Do First

Given a catalogue and two weeks, this order:

One. Pull sixteen months of Search Console query-page-date data. Not the interface, the API, with the date dimension. Everything below depends on it and the interface aggregates away the thing you need.

Two. Run the flipping detection. It needs no keyword tool, no SERP budget, and it finds problems that are already costing money on queries you already rank for. Highest return per hour of anything in this article.

Three. Pull every query where you rank 11–30 with meaningful impressions, sort by impressions, and read the top two hundred. These are the queries Google already thinks you are relevant for. Some need a better page; some need a page that does not exist; some are being served by the wrong page type entirely.

Four. Classify by rules, then read the unclassified bucket by hand. Budget a full day for it. This is where the vocabulary gaps are, and vocabulary gaps are the cheapest wins on the list because they usually need product data rather than new pages.

Five. For your top thirty clusters, look at the SERP. Record the dominant page type. Compare it to the page type you currently have pointed at it. Every mismatch is a decision you have been getting wrong for as long as the page has existed.

Six. Build the map — cluster to URL to page type — before building any page. It will be wrong in places and it will still be the most useful document the project produces, because it makes the disagreements explicit while they are still cheap.

Seven. Fix internal linking against the map before writing anything new. Free, fast, and it resolves a surprising share of what looks like a content problem.

The thread running through all of it is that keyword research on a catalogue is not a discovery exercise. Your customers' vocabulary is already in your Search Console data, your competitors' answers are already in the SERP, and your site's confusions are already visible in which URL Google picked yesterday. The work is reading it, deciding, and writing the decision down somewhere the next person will find it.