SEO

Crawl Trap

Also called spider trap, infinite URL space

A part of a site that can generate URLs without end, so a crawler never reaches the pages that matter.

Quick facts: Crawl Trap

Category
SEO
Also called
spider trap, infinite URL space
Level
Intermediate
Affects
Crawl budget, indexing, index bloat, audit accuracy
Where to see it
Screaming Frog, Sitebulb, server access logs, Search Console Crawl stats
In this article4
  1. How a crawl trap works
  2. Why crawl traps matter
  3. Common mistakes with crawl traps
  4. What to do about it

How a crawl trap works

A crawl trap is any part of a site where following links produces new URLs for as long as anything keeps clicking. Nothing is broken in the ordinary sense — each page returns a valid response and each link goes somewhere real. The problem is that the set of somewheres has no end.

The classic example is an events calendar with a “next month” arrow. The arrow works, so a crawler follows it into next month, and the month after, and on into years that contain nothing. Filters do the same thing by multiplying: colour, then size, then price, then sort order, each combination producing its own address, so a modest catalogue turns into an enormous number of near-identical pages. Session identifiers written into the URL give every visit a fresh copy of the same page. A relative link written wrongly in a template can create paths that repeat a folder over and over, deeper with every hop.

Why crawl traps matter

Crawling is rationed. A search engine decides how much of your site it is worth fetching, and time spent inside a trap is time not spent on the pages you actually sell from. On a small brochure site that may cost you nothing; on a shop with thousands of products it can mean new items sit unindexed for weeks while the crawler works through empty filter combinations. The term for that ration is crawl budget.

The second cost is what reaches the index. Thin, near-duplicate URLs dilute the picture a search engine has of your site, and your own reports get harder to read because the interesting pages are lost in the noise.

Common mistakes with crawl traps

The most common mistake is reaching for robots.txt first. Blocking the pattern stops the fetching, but a blocked URL can still be discovered and can still linger in the index without its content, and it can no longer pass on any signal from links pointing at it. Blocking is a blunt tool and should come after you have decided what those URLs are for.

The opposite mistake is trusting canonical tags to clean it up. A canonical is a hint about which version to prefer, not an instruction to stop crawling, so the crawler still has to fetch every URL to read it.

The third is fixing the symptom in one template and leaving the generator alone. If the code will still mint a new address for every combination someone can click, the trap comes back the next time the theme changes.

What to do about it

Find it first. Run a crawler on your own site and watch for a URL count that keeps climbing, or look at which patterns dominate your server logs. Then decide, pattern by pattern, whether those URLs deserve to exist at all.

Usually the answer is no, and the cleanest fix is at the source: stop generating links to empty calendar months, keep filter combinations behind controls a crawler does not follow, and drop session identifiers out of addresses. Where a URL must exist for people but not for search, use a robots meta tag rather than a block, so the page can still be read and its links still count. On a large shop this is ordinary technical SEO housekeeping, and it usually starts with the rules behind faceted navigation.

Do and do not

Do

  • Crawl your own site and watch the URL count
  • Stop generating links to empty calendar months and filters
  • Keep session identifiers out of URLs entirely

Do not

  • Reach for robots.txt before deciding what the URLs are for
  • Expect canonical tags to stop the crawling
  • Patch one template and leave the generator untouched

Questions people ask about this

How do I know if my site has a crawl trap?

Run a crawler on your own site and let it work. If the number of discovered URLs keeps rising long after you have passed the pages you know exist, something is generating addresses. Server logs tell the same story: one pattern with query strings or repeating folders taking far more crawler requests than your real pages.

Should I block a crawl trap in robots.txt?

It is the fastest brake, not the best repair. A blocked URL can still be discovered from links and can sit in the index without content, and any value from external links pointing at it is lost. Fix the code that mints the URLs where you can, and keep robots.txt for patterns that should never be fetched at all.

Do crawl traps hurt small sites?

Less than large ones, but they are not harmless. A small site rarely runs out of crawl capacity, so the visible damage is index clutter rather than missed pages. It also makes every audit slower and noisier, and traps have a habit of growing when the site does, so it is cheaper to close one early.

Related terms

Found this useful?

Share it, or ask an AI to summarise it

Back to the glossary

Knowing the term is the easy part

Applying it to your own site and budget is the work. Book a call and I will tell you what actually applies to you.