Most site owners believe more indexed pages is better. On large sites, the opposite is usually true. Index bloat, thousands of low-value URLs crawled and indexed, is one of the most common and least understood problems in enterprise SEO, and I have watched it quietly cap sites that were doing everything else right.

How Bloat Happens, and What It Costs

Large sites generate URLs automatically: filtered category views, faceted navigation, search result pages, tag archives, pagination, session parameters, and near-duplicate product variants. Left uncontrolled, Google crawls and indexes tens of thousands of them. On one marketplace I worked on, a huge share of indexed pages should never have been indexed at all. The cost is real and threefold.

The Controls That Fix It

The instinct when traffic is flat is to add more pages. If you have index bloat, that instinct makes the problem worse. The fix is subtraction first: prune the index to the pages that deserve to be there, concentrate crawl budget and authority on them, and only then consider what genuinely new pages are worth adding.

A Real Pattern at Scale

To make this concrete without naming the client: on one global marketplace I worked on, the indexed page count ran into the hundreds of thousands, while the pages that earned meaningful traffic numbered in the low thousands. The gap was faceted navigation and filter combinations, every colour, size, and sort order generating its own indexable URL, plus tag archives and internal search pages Google had happily crawled and stored. The store had not published that content deliberately. The platform generated it, and nobody had told Google to ignore it.

The fix was not more content, it was subtraction. Deciding which filtered views deserved to be landing pages and blocking the rest, canonicalising duplicate paths, and applying noindex to the archives and search pages. Reclaiming crawl budget and concentrating authority on the pages that could actually rank did more for the site than any content campaign of that period. On large sites, the highest-leverage SEO work is often deletion.

How to Run the Diagnosis Yourself

You do not need enterprise tooling to find out whether you have this problem. The check takes an afternoon.

Crawl Budget: How Google Actually Spends Its Time on Your Site

Crawl budget is the amount of crawling Google is willing to do on your site, and on large sites it is finite. It comes down to two forces: how much Google can crawl without straining your servers, and how much it wants to crawl based on how fresh and valuable it judges your pages. Every request spent fetching a filtered URL that will never rank is a request it did not spend on a page that could. On a site of a few hundred pages this is invisible. On a site of hundreds of thousands, it is the ceiling.

You can see the problem in server logs and in Search Console. When a large share of crawl activity lands on parameter URLs, internal search pages, and endless filter combinations, while your genuinely important pages are crawled rarely, crawl budget is being spent on nothing. The symptom owners notice first is different, though: new content that takes weeks to appear in the index, because Google is busy re-crawling junk it was never told to ignore.

The Prune and Consolidate Playbook

Fixing bloat is not one action, it is a sequence, and the order protects you from removing something that was quietly earning. This is the playbook I run, biggest buckets first.

What Recovery Looks Like After a Prune

The first thing that improves is usually not rankings, it is crawl efficiency: important pages get crawled more often and new content gets indexed faster, because the budget is no longer wasted. Rankings follow over the next one to three months as the remaining pages absorb the internal authority once scattered across thousands of dead URLs. On the marketplace I described, reclaiming crawl budget and concentrating authority did more for the site than any content campaign of that period. The lesson holds: on a large site, the highest-leverage SEO work is often what you remove, not what you add.