How a spider trap works
Spiders came first. Before the industry settled on “crawler” and “bot”, the programs that walked the web link by link were called spiders, and a spider trap was a place they walked into and could not walk out of. The name survives in older documentation, in log analysis tools and in the vocabulary of developers who have been doing this a long time. It means exactly the same thing as a crawl trap.
The mechanism is simple. Somewhere on the site, following a link produces a page that contains another link of the same kind, and that one produces another. A calendar that will always render the next month. A filter that will always accept one more combination. A template with a mistaken relative link, so each hop adds another folder to the path and the address grows without ever reaching an error. Every page returns a valid response, so nothing looks wrong from the outside.
Why spider traps matter
A crawler that is stuck is a crawler that is not reading your products, your services or your new article. Search engines allocate a limited amount of fetching to any site and will pull back if the server slows down, so a trap competes directly with the pages you want seen. On a busy shop that shows up as new items taking a long time to appear in search.
There is a hosting cost too, and it lands hardest on small plans. Every trapped request is a real request your server has to answer, and shared hosting is where that bill arrives first — as slower pages for genuine visitors, long before anyone notices the pattern.
Common mistakes with spider traps
The first mistake is trusting the response. A trapped URL answers normally, so a status check finds nothing to report, and the page looks fine when you open it. Valid does not mean wanted, and that is the whole difficulty with this problem.
The second is blaming the tool. Someone starts a crawl to audit the site, comes back later to a URL count still climbing, and assumes the site is simply large. It is worth pausing there: if the new addresses are variations of one page with different dates, parameters or repeating folders, your own crawler has found the trap for you. Log file analysis confirms it, because one shape of address will be taking a large share of the crawler requests while your commercial pages are visited rarely.
The third is treating index bloat as a separate issue. Far more URLs known to search engines than you have real pages is often the same trap seen from the other end.
What to do about it
Identify the pattern before you touch anything, because the fix depends on why those URLs exist. If nobody needs them, stop the site from creating them: no links into empty calendar months, no filter combinations exposed as crawlable links, no session identifiers written into addresses.
If people do need them but search engines do not, a robots meta tag on the page is usually better than a blanket block, because the page can still be read and its links still carry weight. Whichever route you take, crawl again afterwards and confirm the count settles. A trap that has only been narrowed is still a trap.
