SEO

Spider Trap

Also called crawl trap, infinite crawl space

The older name for a crawl trap — a structure that keeps a crawler moving through URLs that never run out.

Quick facts: Spider Trap

Category
SEO
Also called
crawl trap, infinite crawl space
Level
Intermediate
Affects
Crawl budget, indexing, server load, audit time
Where to see it
Screaming Frog, Sitebulb, Screaming Frog Log File Analyser, server access logs
In this article4
  1. How a spider trap works
  2. Why spider traps matter
  3. Common mistakes with spider traps
  4. What to do about it

How a spider trap works

Spiders came first. Before the industry settled on “crawler” and “bot”, the programs that walked the web link by link were called spiders, and a spider trap was a place they walked into and could not walk out of. The name survives in older documentation, in log analysis tools and in the vocabulary of developers who have been doing this a long time. It means exactly the same thing as a crawl trap.

The mechanism is simple. Somewhere on the site, following a link produces a page that contains another link of the same kind, and that one produces another. A calendar that will always render the next month. A filter that will always accept one more combination. A template with a mistaken relative link, so each hop adds another folder to the path and the address grows without ever reaching an error. Every page returns a valid response, so nothing looks wrong from the outside.

Why spider traps matter

A crawler that is stuck is a crawler that is not reading your products, your services or your new article. Search engines allocate a limited amount of fetching to any site and will pull back if the server slows down, so a trap competes directly with the pages you want seen. On a busy shop that shows up as new items taking a long time to appear in search.

There is a hosting cost too, and it lands hardest on small plans. Every trapped request is a real request your server has to answer, and shared hosting is where that bill arrives first — as slower pages for genuine visitors, long before anyone notices the pattern.

Common mistakes with spider traps

The first mistake is trusting the response. A trapped URL answers normally, so a status check finds nothing to report, and the page looks fine when you open it. Valid does not mean wanted, and that is the whole difficulty with this problem.

The second is blaming the tool. Someone starts a crawl to audit the site, comes back later to a URL count still climbing, and assumes the site is simply large. It is worth pausing there: if the new addresses are variations of one page with different dates, parameters or repeating folders, your own crawler has found the trap for you. Log file analysis confirms it, because one shape of address will be taking a large share of the crawler requests while your commercial pages are visited rarely.

The third is treating index bloat as a separate issue. Far more URLs known to search engines than you have real pages is often the same trap seen from the other end.

What to do about it

Identify the pattern before you touch anything, because the fix depends on why those URLs exist. If nobody needs them, stop the site from creating them: no links into empty calendar months, no filter combinations exposed as crawlable links, no session identifiers written into addresses.

If people do need them but search engines do not, a robots meta tag on the page is usually better than a blanket block, because the page can still be read and its links still carry weight. Whichever route you take, crawl again afterwards and confirm the count settles. A trap that has only been narrowed is still a trap.

Do and do not

Do

  • Treat a crawl that never finishes as a warning sign
  • Group crawler requests by URL pattern in your logs
  • Crawl again after the fix and confirm the count settles

Do not

  • Assume a valid response means the URL should exist
  • Block the pattern before you know who needs it
  • Ignore the server load it puts on shared hosting

Questions people ask about this

Is a spider trap different from a crawl trap?

No, they are two names for the same thing. Spider was the older word for the program that follows links across the web, and it survives in log analysis tools and in documentation written years ago. If a developer mentions a spider trap and an SEO report mentions a crawl trap, they are describing the same structure.

Can a spider trap get my site penalised?

Not as a penalty in itself. The damage is practical rather than punitive: crawling is spent on pages that do not matter, useful pages are visited less often, and thin near-duplicate URLs clutter the index. Search engines simply crawl less of what you want seen, which looks like a ranking problem but is a structural one.

My crawler never finishes. Is that always a trap?

Not always, but it is the first thing to check. A very large site genuinely takes a long time, so compare the URLs being discovered against the pages you know exist. If the new addresses are variations of the same page with different parameters, dates or repeating folders, you have found a trap rather than a big site.

Related terms

Found this useful?

Share it, or ask an AI to summarise it

Back to the glossary

Knowing the term is the easy part

Applying it to your own site and budget is the work. Book a call and I will tell you what actually applies to you.