Progressive Web Apps (PWA) for SEO & Performance

1. Twelve Thousand Products, Four Hundred Indexed

In March 2024 a tools and fixings distributor called me about a traffic collapse. They had relaunched in November on a React storefront — a progressive web app, service worker, install prompt, the full set — and organic sessions had fallen 71% over the following four months. The agency that built it told them Google needed time to adjust to the new site.

Search Console said something more specific. Of 12,400 product URLs in the sitemap, 431 were indexed. Another 6,800 sat in "Crawled — currently not indexed" and about 5,000 in "Discovered — currently not indexed", which means Google had seen the URL and had not even bothered to fetch it.

I ran one command:

# What does the server actually return, before any JavaScript runs?
curl -sL https://example.com/products/m8-hex-bolt-zinc-100 \
  | grep -c "M8 Hex"
# 0

Zero. The product name did not exist in the HTML the server sent. Neither did the price, the description, the specification table, or a single internal link to another product. The document was a nav, a footer, an empty <div id="root">, and 640KB of JavaScript that would eventually build the page in the browser.

Google can render JavaScript. That is true and it gets repeated as though it settles the matter. What it does not tell you is that rendering is a separate, queued, resource-limited stage that behaves nothing like the initial crawl, and that a catalogue of twelve thousand pages entirely dependent on it will be indexed slowly, partially, and unpredictably. That is what happened here, and it is what this article is about.

This is the crawlability half of the PWA story. The business and experience case — install prompts, push, whether any of it is worth building — is in the companion piece on PWAs for ecommerce, and the caching mechanics are in the service workers article. Here I am only concerned with one question: can a search engine see your catalogue, and how do you prove it.

2. Where the Risk Actually Comes From

Let me be precise about the causation, because "PWAs are bad for SEO" is both a common belief and wrong.

Nothing in the definition of a progressive web app is a crawling problem. A manifest is a JSON file crawlers ignore. A service worker is never executed by Googlebot at all — each crawl is a fresh browser profile with no registration, so whatever your worker does is invisible to indexing. HTTPS is a positive. You can add all three to a perfectly ordinary server-rendered storefront tomorrow and your indexing will not move a millimetre.

The risk arrives because of what usually rides along. PWA projects are, in practice, rewrites. Somebody decides that app-like navigation requires a client-side router, the router requires JSON endpoints instead of HTML documents, and within a sprint the catalogue exists only as data assembled in the browser. The manifest gets the blame in the post-mortem because it is what changed in the project name.

So the useful framing: the PWA parts are neutral, and the single-page architecture that often accompanies them is where every problem in this article originates. If you are adding a manifest and a service worker to an existing server-rendered site, you can stop reading and go do that; it is safe. If someone is proposing a client-side rewrite and calling it a PWA, the rest of this is for you.

3. What Googlebot Actually Does With JavaScript

The model that matters is a pipeline with a queue in the middle.

Googlebot fetches a URL and gets the raw HTML response. It parses that for links, for meta directives, for canonical tags, and for content. If the page needs JavaScript to be meaningful, the URL is placed into a render queue. Some time later — and the time is the whole issue — a headless Chromium instance fetches the page, executes the JavaScript, waits for network activity to settle, and produces a rendered DOM. That rendered DOM is then indexed, and any new links discovered in it go back into the crawl queue.

Several properties of that pipeline cause trouble on a large catalogue.

The queue has a delay and it is not fixed. Google has said the median is seconds and I believe them for a well-known site with a small page count. What I measure on mid-market ecommerce is different: new product URLs on a client-rendered site typically appear in the index days after their server-rendered equivalents, and on the distributor above the gap was averaging eleven days across the sample I checked. The median is not the number that hurts you; the tail is.

Rendering is rate-limited by your crawl budget in a way initial fetching is not. Rendering a page costs Google roughly an order of magnitude more than fetching it. On a site with twelve thousand product URLs plus filtered category permutations, that cost is what decides how much gets rendered at all. This is the mechanism behind "Discovered — currently not indexed" at scale: Google has the URL and has decided rendering it is not worth the compute right now.

Links only discovered after rendering are discovered late. If your category pages need JavaScript to list their products, the product URLs are not found in the crawl stage. They are found in the render stage, which means the discovery of your catalogue is gated behind the most expensive part of the pipeline. On a server-rendered site the whole catalogue is discoverable in one cheap pass.

Rendering can fail silently. A JavaScript error, a request to an API that times out, a third-party script that hangs — any of these can produce a rendered DOM that is missing your content, and there is no error report. The page simply gets indexed as whatever managed to render, which is often the nav and the footer.

That last one produces the most confusing outcome on this list: pages indexed with no content, ranking for nothing, with Search Console reporting them as perfectly fine.

4. What "Critical Content in the Server HTML" Means Precisely

Everyone nods at this advice and then implements it partially. Here is the list I actually check, in priority order, for a product page.

The <title> and meta description. Not injected by a helmet library after hydration — present in the response body. A title written by JavaScript is a title that exists only for the render stage, and it also means every social scraper and link preview gets your generic fallback.

The canonical tag. This one is serious enough to have its own section below.

The H1 and the product name. Obvious and routinely missing.

Price and availability, as text. Not only in the structured data. If the visible price comes from a client-side API call, the rendered page Google indexes may show a loading skeleton, and skeleton text is what gets extracted.

The full product description and specifications. Including anything behind a tab or an accordion. Content hidden in a collapsed panel is fine as long as it is in the DOM; content fetched when the panel is opened is not.

Product schema in the response. A JSON-LD block written by JavaScript will usually be picked up in rendering, but it is one more thing that depends on the fragile stage, and the Rich Results test will pass while real indexing is inconsistent.

Real <a href> elements to related products, the parent category, and breadcrumbs. This is the one that governs crawl efficiency for the whole site.

A quick way to check the whole set at once:

#!/usr/bin/env bash
# Check a URL's raw HTML for the things that must not require JavaScript.
URL="$1"
HTML=$(curl -sL -A "Mozilla/5.0 (compatible; Googlebot/2.1)" "$URL")

check() { printf '%-22s %s\n' "$1" "$(grep -qi -- "$2" <<< "$HTML" && echo OK || echo MISSING)"; }

check "title"        "<title"
check "canonical"    "rel=\"canonical\""
check "h1"           "<h1"
check "product schema" "\"@type\": *\"Product\""
check "price text"   "£"

# How many real links does the raw document contain?
echo "internal links: $(grep -o 'href="/[^"]*"' <<< "$HTML" | sort -u | wc -l)"

On the distributor's product pages that script reported a title, no canonical, no H1, no schema, no price, and four internal links — all of them in the header nav. On a healthy product page I expect somewhere north of forty unique internal links in the raw HTML, most of them to sibling products and category facets.

5. The curl Test, and Where It Misleads You

Fetching raw HTML is the fastest diagnostic available and it answers exactly one question: what is in the initial response. It does not tell you what Google indexed. Three ways people over-read it.

First, a missing element in the raw HTML is not proof the page will not be indexed correctly. Google may well render it fine. What you have found is a dependency on the fragile stage, not a confirmed failure. The correct conclusion is "this is at risk", and the confirmation comes from the index.

Second, present in the raw HTML is not proof it survives. I have seen a server-rendered canonical overwritten by a client-side routing library on hydration, and the rendered DOM is what Google uses. Server-side presence is necessary, not sufficient.

Third, your curl request is not a Googlebot request. Some CDNs and bot-management layers serve different responses by user agent, and a few serve a prerendered variant to crawlers. If a client swears their pages are server-rendered and curl disagrees, check whether something is fronting the origin before accusing anyone.

The tool that answers the real question is the URL Inspection tool in Search Console, specifically "Test live URL" followed by "View tested page". That gives you the rendered HTML Google's own renderer produced, a screenshot, and — the most useful panel and the one nobody opens — the list of page resources that failed to load. Half the rendering bugs I have diagnosed were visible in that list as a blocked or timed-out request.

A local approximation that catches most problems before you deploy:

// render-check.js — compare raw HTML against the rendered DOM.
import puppeteer from 'puppeteer';

const url = process.argv[2];
const raw = await (await fetch(url)).text();

const browser = await puppeteer.launch();
const page = await browser.newPage();
// Approximate Googlebot: mobile viewport, no service worker, and a
// hard cap on how long we are willing to wait for the network.
await page.setUserAgent('Mozilla/5.0 (Linux; Android 6.0.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Mobile Safari/537.36 (compatible; Googlebot/2.1)');
await page.setViewport({ width: 412, height: 892, isMobile: true });
await page.goto(url, { waitUntil: 'networkidle0', timeout: 20000 });

const rendered = await page.content();
const title = await page.title();
const h1 = await page.$eval('h1', (el) => el.textContent).catch(() => null);
const canonical = await page
  .$eval('link[rel=canonical]', (el) => el.href)
  .catch(() => null);

console.log({
  rawBytes: raw.length,
  renderedBytes: rendered.length,
  // A large ratio means most of the page only exists after JavaScript.
  ratio: (rendered.length / raw.length).toFixed(1),
  titleInRaw: raw.includes(title),
  h1InRaw: h1 ? raw.includes(h1.trim()) : false,
  canonicalInRaw: canonical ? raw.includes(canonical) : false
});

await browser.close();

The ratio is the number I watch. A healthy server-rendered product page comes out between 1.0 and 1.4. The distributor's pages were at 34. Anything over about 3 means the majority of your page exists only after execution, and you should assume everything in this article applies to you.

6. The Canonical Tag Problem

Canonicals deserve separating out because the failure is quiet and the consequences are wide.

If a canonical tag is written by JavaScript, it exists only in the rendered DOM. Google generally picks it up, but the initial crawl sees no canonical at all and may act on that — particularly during the window before rendering happens. On a site with filtered category URLs, that window is when duplicate URLs get into the index.

The failure I see most often is worse and more specific: a routing library sets the canonical to the current URL on every navigation, including URLs that should canonicalise elsewhere. Filter and sort parameters are the usual victim. A category page at /screws?sort=price&page=3 ends up self-canonical, and now you have several thousand near-duplicate URLs each declaring itself the original.

On the distributor this had produced roughly 41,000 indexed URLs against 12,400 real products, which is a large part of why the genuine pages were not getting rendered — the crawl budget was being spent on permutations.

<!-- Server-rendered, computed from the canonical route,
     with tracking and pagination parameters stripped. -->
<link rel="canonical" href="https://example.com/c/screws-fixings/wood-screws" />

<!-- And for the filtered variants that should not be indexed
     but should still be crawlable for discovery. -->
<meta name="robots" content="noindex,follow" />

The rule I apply: canonical, robots meta, and hreflang are server responsibilities. If your framework offers a component that writes them client-side, use it for nothing that matters. And if you are on a headless setup where the frontend genuinely does own the head, make the canonical a server-computed value passed into the initial render rather than something derived from window.location, which is a bug waiting for its first tracking parameter. The same applies to any headless commerce architecture — the broader trade-offs there are covered in the piece on headless commerce SEO and performance.

7. Soft 404s, and the Status Code a Router Cannot Send

This is the single most common client-side routing bug on ecommerce sites and it is structural rather than accidental.

A client-side router receives a URL that matches no route. It renders a "Product not found" component. The customer sees a sensible message. Googlebot sees an HTTP 200 response containing a page, because the server has already answered 200 with the app shell before the router ever ran. There is no mechanism by which JavaScript can change a status code that has already been sent.

This matters because discontinued products are constant on a catalogue site. Every delisted SKU becomes a 200-status page with almost no content and a heading that says the product is unavailable. Google's classifier eventually flags these as soft 404s, but "eventually" can be months, and in the meantime they consume crawl budget and dilute the site's quality signals.

On the distributor there were about 2,900 of these. Their categories included discontinued lines that still had links pointing at them, so the crawler kept finding them, kept fetching them, kept rendering them, and kept getting a 200 with a sad message.

There are three fixes, in descending order of how much I like them.

Handle unknown routes on the server. If you have any server rendering — Next.js, Nuxt, Remix, a Node layer in front of a static build — the route resolver knows whether the SKU exists before it responds, so it can send a real 404 or a 301 to the category. This is the correct fix and it is available on more architectures than people assume.

// Next.js app router — a real 404 status for an unknown product.
export default async function ProductPage({ params }) {
  const product = await getProduct(params.sku);
  // notFound() renders the not-found UI *and* sends HTTP 404.
  if (!product) notFound();
  if (product.replacedBy) {
    // Discontinued but superseded: send them to the successor.
    redirect(`/products/${product.replacedBy}`, 301);
  }
  return <ProductView product={product} />;
}

Serve a noindex header for unknown paths at the edge. If the routing is genuinely client-only, a CDN worker can check the URL against a list of valid SKUs and add X-Robots-Tag: noindex to responses for anything unmatched. It is not as good as a 404 but it stops the pages entering the index.

// Cloudflare Worker — mark unknown product URLs as noindex.
export default {
  async fetch(request, env) {
    const url = new URL(request.url);
    const response = await fetch(request);

    if (url.pathname.startsWith('/products/')) {
      const sku = url.pathname.split('/')[2];
      // KV namespace populated by the catalogue export on each build.
      const exists = await env.SKUS.get(sku);
      if (!exists) {
        const r = new Response(response.body, response);
        r.headers.set('X-Robots-Tag', 'noindex');
        return r;
      }
    }
    return response;
  }
};

Redirect client-side to a real 404 URL. The worst option: the router pushes the browser to /404, which the server does answer with a 404. It works, sort of, and it is a redirect chain and a flash of the wrong content. I would take it over nothing.

The same problem applies to every other status code a storefront needs. Out of stock should not be a 404. A retired category should be a 301. A geo-restricted product should probably be a 404 in that region rather than a page saying the customer cannot buy it. All of these are server decisions, and a client-only architecture has quietly given away the ability to make them.

8. Links That Are Not Links

Google finds pages by following href attributes on anchor elements in the DOM. That sentence is doing a lot of work and there is a family of common patterns that fail it.

A <div> with an onclick handler that calls the router is not a link. A <button> that navigates is not a link. An anchor with href="#" and a JavaScript handler is not a link. A router component that renders correctly is fine — most do emit real anchors — but I have reviewed enough custom implementations that emit spans to check every time rather than assume.

# Count anchors with a real href in the rendered DOM, and how many
# clickable-looking elements are not anchors at all.
node -e "
const p = require('puppeteer');
(async () => {
  const b = await p.launch();
  const pg = await b.newPage();
  await pg.goto(process.argv[1], { waitUntil: 'networkidle0' });
  console.log(await pg.evaluate(() => ({
    realLinks: document.querySelectorAll('a[href]:not([href^=\"#\"])').length,
    fakeLinks: document.querySelectorAll('[onclick], [role=link]:not(a)').length
  })));
  await b.close();
})();
" "$1"

The second pattern that costs discovery is infinite scroll with no underlying pagination. A category page that loads twenty products and fetches more on scroll exposes exactly twenty product links to a crawler, because Googlebot does not scroll. Whatever is on page four of your best category is, as far as discovery is concerned, not linked from anywhere.

The fix is not to remove infinite scroll. It is to have real paginated URLs underneath it — /c/wood-screws?page=2 — that are server-rendered, contain real links, and are reachable via anchors in the page even if those anchors are visually subordinate to the scrolling behaviour. Customers get the scroll, crawlers get the pagination, and you get a set of URLs you can actually inspect when something goes wrong.

A third, subtler discovery problem: a well-built SPA often stops requesting the server for navigation entirely, which means your server logs stop showing page views. Crawl analysis from log files is one of the better diagnostic techniques available on a large catalogue, and a client-side router removes the data it depends on for real users. It does not affect Googlebot, which always makes real requests, but it does mean your logs no longer tell you what humans and crawlers are doing on comparable terms.

9. Rendering Strategies, Ranked for a Catalogue

The choice is not binary and the middle options are where most storefronts should sit.

Two rows deserve comment.

"SSR shell plus client-fetched data" is the trap. It looks server-rendered — you get HTML back, the layout is there, the nav is there — and the price, stock, and description arrive from an API call after load. Teams believe they have solved the problem because curl returns real markup. The content that determines whether the page ranks is still on the fragile stage. If you check one thing after reading this article, check whether your product data is in the response or fetched afterwards.

Dynamic rendering — detecting crawlers and serving them a prerendered snapshot — was Google's own recommendation for years and was downgraded to a workaround in 2022. It works. It is also a second rendering system that will drift out of sync with the real one, and the drift is invisible until someone notices the snapshot service has been serving a cached version from six weeks ago. I have inherited exactly that. Use it to stop bleeding while you fix the architecture, not as the architecture.

The strategy I recommend most often for large catalogues is pre-rendering with incremental regeneration, which gets you static HTML for crawlers and a manageable build. Its own failure modes — build times, stale pages, where dynamic commerce breaks the static model — are the subject of the article on JAMstack ecommerce.

10. Rendering Delay, Measured

"How long does Google take to render?" gets answered with anecdote. Here is how to measure it on your own site, which is the only number that should influence your decisions.

Publish a new product with a unique, invented token in the body — something like a nonsense string that appears nowhere else on the web. Note the timestamp. Then poll the index for that token.

# Check whether the token has been indexed yet.
# Run daily; the first day it returns a result is your indexing lag.
TOKEN="zx7qwelm"
curl -s "https://www.google.com/search?q=%22$TOKEN%22+site:example.com" \
  -A "Mozilla/5.0" | grep -qi "$TOKEN" && echo "indexed" || echo "not yet"

Do this for two variants: one token in the server HTML, one injected by JavaScript only. The difference between the two dates is your render lag, on your site, at your current crawl rate. It is the most useful single measurement in this whole area and almost nobody takes it.

On the distributor, before the fix, the server-HTML token was indexed in 3 days and the JavaScript-only token in 19. After they moved product pages to server rendering, the server token was indexed in under 48 hours. That was the evidence that made the case internally, and it was worth more than any amount of argument about how capable Googlebot is in principle.

A caveat on the method: scraping search results is fragile and against the terms of service if you do it at volume. Run it a handful of times a day, manually if you prefer, or use the URL Inspection API which is the supported route and gives you the last crawl date directly.

11. The Service Worker Does Not Index Anything

Worth stating flatly because it comes up in every one of these projects: Googlebot does not register or execute service workers. Every crawl is a first visit with an empty cache. Nothing your worker does — good or bad — affects what gets indexed.

This has two practical consequences.

You cannot use a service worker to fix a rendering problem. Serving crawlers a cached, complete version of a page is not possible, and if it were it would be cloaking.

You also cannot break indexing with one, which is more reassuring than it sounds. The worst service worker bug I have seen — eleven days of a stale promotional price — was invisible to search entirely. The customers saw it; Google never did.

The one genuine, indirect effect is on field performance data. Cached assets make repeat visits faster, repeat visits are part of the Chrome User Experience Report, and so your Core Web Vitals field data improves modestly. On the sites where I have measured it, the 75th percentile LCP moved by 8-15% after a service worker rollout, which is real but is not going to rescue a page that is failing for other reasons. And it is worth remembering that the same caching hides errors from your monitoring: customers served from cache never hit your origin, so your error rates can look healthy while a slice of your audience sees something broken.

12. What the Caching Does and Does Not Do for Core Web Vitals

Since this is filed under performance, let me be specific about which metrics move.

LCP on repeat visits: improves, substantially. Fonts, CSS, and the hero image come from local storage. This is the biggest effect and it is confined to visits two and onward.

LCP on first visits: unchanged, or slightly worse. The worker installs during that visit and competes for bandwidth. If you pre-cache a long list of assets, measurably worse — which is why I pre-cache almost nothing.

INP: unchanged by caching, and often made worse by the SPA architecture. Interaction latency is about main-thread work, and a client-side router adds main-thread work. If you are moving to an SPA to feel faster, INP is the metric that will tell you whether you succeeded, and in my experience it usually gets worse before it gets better.

CLS: unchanged by caching, at risk from install banners. An install prompt that inserts itself into the flow after load is a layout shift. Anchor it with position: fixed and it is free.

TTFB: unchanged for uncached navigations, near-zero for cached ones. Which distorts your averages pleasantly and your understanding unhelpfully, so segment your field data by whether a service worker was controlling the page. The broader measurement discipline is covered in measuring Core Web Vitals in CI.

The summary I give clients: a service worker is a repeat-visit optimisation wearing a performance costume. If your problem is first-visit speed — and on most storefronts it is, because most sessions are first visits — it will not help you.

13. The Distributor, Fixed

What we actually did, over about five months, and what it cost.

Weeks one to three: stop the bleeding. Server-rendered canonicals via an edge worker, computed from a path-matching table, stripping all query parameters except page. X-Robots-Tag: noindex on any URL with a filter parameter. This removed nothing from the index immediately but it stopped the growth, and indexed URLs began falling from 41,000 within a fortnight.

Weeks two to six, in parallel: the 404 problem. An edge worker checking SKUs against a KV store refreshed on each catalogue export, returning a genuine 404 for unknown products and a 301 for the 400-odd that had a documented successor. About 2,900 soft 404s stopped being fetched.

Weeks four to fourteen: server rendering for product and category pages. The expensive part. They kept React, moved to Next.js with the app router, and rendered product and category routes on the server with the data in the initial payload. Everything behind login stayed client-rendered, which is fine because none of it should be indexed.

Weeks twelve to sixteen: real pagination under the infinite scroll. Category pages got server-rendered paginated URLs with anchor links, and the scroll behaviour was layered on top.

Results at month six from the start of work: 11,900 of 12,400 product URLs indexed, against 431 at the start. Organic sessions recovered to 94% of the pre-relaunch baseline at month seven and passed it at month nine. Render lag on a test token fell from 19 days to under 2.

What went wrong. Two things worth recording.

The edge worker doing canonical rewriting had a path-matching bug that affected category URLs containing a hyphenated two-word segment — roughly 6% of categories — and pointed their canonicals at the wrong parent. Nobody noticed for five weeks because the affected pages were low traffic and the aggregate numbers were improving. When we found it, those categories had been consolidated into their wrong parents in the index and took another two months to recover. The lesson I took: when you deploy a canonical rewrite, sample fifty URLs across every path shape you have and check each one by hand, before deploying. It takes an hour.

The second was a hydration mismatch on the price element. The server rendered the list price and the client hydrated with the customer-specific trade price, and React's mismatch handling produced a brief flash and, on some pages, a discrepancy between the rendered price and the price in the JSON-LD. Search Console flagged it as a structured data mismatch about three weeks later. It was fixed by rendering the anonymous price server-side and only applying trade pricing after an explicit login check, but I would now treat any price that varies by customer as something to keep out of the initial render entirely.

And one thing I would do differently at a strategic level: we should have run the token-indexing test in week one rather than week six. It is a two-day setup and it would have given us the single most persuasive number in the project before we had to argue for a fourteen-week rebuild.

14. Monitoring It Continuously

Rendering regressions are silent. A dependency upgrade moves a component from server to client, nobody notices, and three months later the traffic is down. The defence is a check in the pipeline.

# .github/workflows/render-check.yml
# Fail the build if critical content stops appearing in the server HTML.
name: SEO render check
on: [pull_request]

jobs:
  check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm ci && npm run build && npm run start &
      - run: npx wait-on http://localhost:3000
      - run: node scripts/assert-server-html.mjs

// scripts/assert-server-html.mjs
// Fetch a representative URL of each template and assert the
// non-negotiables are present before any JavaScript runs.
const CASES = [
  { url: '/products/m8-hex-bolt-zinc-100', needs: ['<h1', 'rel="canonical"', '"@type":"Product"', 'M8 Hex'] },
  { url: '/c/screws-fixings',              needs: ['<h1', 'rel="canonical"', 'href="/products/'] },
  { url: '/blog/torque-settings-guide',    needs: ['<h1', 'rel="canonical"'] }
];

let failed = false;
for (const { url, needs } of CASES) {
  const html = await (await fetch('http://localhost:3000' + url)).text();
  for (const needle of needs) {
    if (!html.includes(needle)) {
      console.error(`FAIL ${url}: missing ${needle}`);
      failed = true;
    }
  }
  // A page that is mostly script tags is a page that mostly is not there.
  const scriptBytes = [...html.matchAll(/<script[\s\S]*?<\/script>/g)]
    .reduce((n, m) => n + m[0].length, 0);
  if (scriptBytes / html.length > 0.7) {
    console.error(`FAIL ${url}: ${Math.round(scriptBytes / html.length * 100)}% script`);
    failed = true;
  }
}
process.exit(failed ? 1 : 0);

Alongside that, two things worth watching weekly in Search Console: the count in "Crawled — currently not indexed" as a trend rather than an absolute, and the ratio of indexed pages to sitemap pages per template. A drop in one template's ratio is the earliest signal you will get that a rendering change has broken something, usually weeks before traffic moves.

And check the rendered screenshot in URL Inspection after any significant frontend release. It takes thirty seconds and it has caught two problems for me that no automated check would have — once a cookie consent overlay covering the entire rendered page, and once a geo-redirect that sent the renderer to a country selector.

15. Blocking the Resources That Build Your Page

An old mistake that keeps coming back, and it is nastier on a client-rendered site than it ever was on a server-rendered one.

If robots.txt disallows the path your JavaScript bundles are served from, Googlebot cannot fetch them, so rendering produces an empty shell. On a server-rendered site that costs you some interactivity in the rendered screenshot and nothing else. On a client-rendered site it costs you the entire page. The content is not merely degraded; it does not exist.

The reason this survives in the wild is that the disallow rules are usually old and were written for good reasons. Someone blocked /static/ in 2016 to stop the crawler wasting budget on assets, which was reasonable advice at the time, and then five years later the frontend moved to a bundler that emits into that directory.

# robots.txt — the rules that break rendering, and what to do instead

# Wrong: blocks the bundle that builds every page.
Disallow: /_next/
Disallow: /static/
Disallow: /assets/

# Right: allow anything the renderer needs, block only what
# generates crawl waste and has no bearing on rendering.
Allow: /_next/static/
Disallow: /search
Disallow: /cart
Disallow: /*?*sort=
Disallow: /*?*utm_

The same applies to any API endpoint your page calls during render. If your product page fetches /api/pdp/{sku} and /api/ is disallowed, the render will produce a skeleton. I have found this twice, both times on sites where somebody had blocked the API path for what they described as security reasons — which it is not, since a disallow is a request to well-behaved crawlers and no obstacle at all to anyone hostile.

Third-party resources matter too, and are easier to miss. A page that cannot render without a script from a tag manager, a personalisation vendor, or a font service is a page whose rendering depends on somebody else's uptime and somebody else's robots rules. The blocked-resources panel in the URL Inspection tool lists all of it, which is why I keep recommending that panel over every other diagnostic in Search Console.

One more, specific to storefronts with geographic routing: if your edge redirects visitors by IP to a country-specific path, remember that Googlebot crawls predominantly from US addresses. A UK retailer who redirects every non-UK IP to a "we do not ship to your country" page has just told the crawler that is what the whole site is. Serve crawlers the canonical locale, or better, do not IP-redirect at all and offer a country switcher instead.

16. Things That Are Not Actually Problems

An equal and opposite failure mode is being so cautious about JavaScript that you cripple the site. A few things people worry about that they should not.

JavaScript-driven filters and sorting on a category page. Fine, as long as the default unfiltered view is server-rendered and the filtered states have crawlable URLs where you want them indexed and are noindexed where you do not.

Lazy-loaded images. Fine with native loading="lazy". Googlebot handles it. Custom scroll-driven implementations that only set src on intersection are a different matter, because the renderer does not scroll — but native lazy loading is not that.

Content in tabs and accordions. Fine if it is in the DOM. Google has been clear that hidden-by-default content is indexed and, since the mobile-first index, treated at full weight.

React, Vue, or Svelte as such. The framework is irrelevant. Where the rendering happens is the entire question. A Next.js site rendering on the server is as crawlable as a PHP one, and a PHP site that outputs an empty div and a bundle is as broken as any SPA.

A large JavaScript bundle, from a crawling perspective. It is a performance problem and a user problem, and Googlebot has more patience than your customers do. Do not conflate the two arguments; fix the bundle for the humans.

17. Questions That Come Up

"Google says it renders JavaScript. Why is this still a problem?" Because rendering is queued, rate-limited, and can fail without telling you, and because links discovered only after rendering are discovered late. On a fifty-page site none of that matters. On a twelve-thousand-page catalogue it decides how much of your site exists.

"Is prerendering for bots cloaking?" Not if the content is equivalent. It is a supported workaround and Google has said so. The practical risk is drift: the snapshot service falls behind and starts serving crawlers a different site than the one users get, and at that point it is cloaking whether you meant it or not.

"Do we need to server-render everything?" No. Server-render anything you want indexed, which is the catalogue, the content, and the landing pages. Account pages, order history, the wishlist, the checkout — all of that should be client-rendered and noindexed, and you will be glad of the simplicity.

"Our Lighthouse SEO score is 100. Are we fine?" No. Lighthouse runs JavaScript, so it sees the rendered page and reports the title and meta description it finds there. It cannot tell you whether those existed in the server response. A perfect Lighthouse SEO score is compatible with a catalogue that is entirely invisible to the initial crawl.

"How long after fixing this should we expect recovery?" On the sites I have done this on, indexing counts move within two to four weeks and traffic follows over two to four months. If you also had a duplicate URL problem, add time — consolidating a bloated index is slower than adding pages to it.

"Should we submit a sitemap with lastmod to speed it up?" Yes, and make lastmod honest. Google has said it uses the field and also that it ignores it from sites where it is obviously wrong. A sitemap where every URL claims to have been modified today is a sitemap Google stops trusting.

"Does the install prompt or manifest affect any of this?" Not at all. Add them freely. It is the rendering architecture that matters, and the two get bundled together by project scope rather than by any technical necessity.

18. What I Would Do First

In order.

One. Run curl against one product page and one category page and count how many of the seven critical elements are in the response. Ten minutes, no tooling, and it tells you immediately whether you have a problem or not.

Two. Open Search Console and compare indexed URLs against sitemap URLs, per template. If products are indexed at under 80% of their sitemap count, you have a discovery or rendering problem regardless of what the traffic graph says this month.

Three. Set up the two-token indexing lag test. It takes a day and it gives you the number that will settle every subsequent argument.

Four. Fix canonicals and status codes before touching the rendering architecture. They are cheap, they can usually be done at the edge without a rebuild, and they stop the index getting worse while you plan the expensive work.

Five. Move product and category templates to server rendering. Nothing else. Leave the account area, the cart, and anything behind login exactly where it is.

Six. Put the assertion script in CI so the next person who moves a component to the client finds out in a pull request rather than in a traffic report.

The distributor's agency was not lying when they said Google renders JavaScript. They were answering a different question from the one that mattered. The question that mattered was whether Google would render twelve thousand pages of low-authority product content, promptly, and reliably, and the answer to that was visible in four months of Search Console data before anyone thought to look.