How a crawl trap works
A crawl trap is any part of a site where following links produces new URLs for as long as anything keeps clicking. Nothing is broken in the ordinary sense — each page returns a valid response and each link goes somewhere real. The problem is that the set of somewheres has no end.
The classic example is an events calendar with a “next month” arrow. The arrow works, so a crawler follows it into next month, and the month after, and on into years that contain nothing. Filters do the same thing by multiplying: colour, then size, then price, then sort order, each combination producing its own address, so a modest catalogue turns into an enormous number of near-identical pages. Session identifiers written into the URL give every visit a fresh copy of the same page. A relative link written wrongly in a template can create paths that repeat a folder over and over, deeper with every hop.
Why crawl traps matter
Crawling is rationed. A search engine decides how much of your site it is worth fetching, and time spent inside a trap is time not spent on the pages you actually sell from. On a small brochure site that may cost you nothing; on a shop with thousands of products it can mean new items sit unindexed for weeks while the crawler works through empty filter combinations. The term for that ration is crawl budget.
The second cost is what reaches the index. Thin, near-duplicate URLs dilute the picture a search engine has of your site, and your own reports get harder to read because the interesting pages are lost in the noise.
Common mistakes with crawl traps
The most common mistake is reaching for robots.txt first. Blocking the pattern stops the fetching, but a blocked URL can still be discovered and can still linger in the index without its content, and it can no longer pass on any signal from links pointing at it. Blocking is a blunt tool and should come after you have decided what those URLs are for.
The opposite mistake is trusting canonical tags to clean it up. A canonical is a hint about which version to prefer, not an instruction to stop crawling, so the crawler still has to fetch every URL to read it.
The third is fixing the symptom in one template and leaving the generator alone. If the code will still mint a new address for every combination someone can click, the trap comes back the next time the theme changes.
What to do about it
Find it first. Run a crawler on your own site and watch for a URL count that keeps climbing, or look at which patterns dominate your server logs. Then decide, pattern by pattern, whether those URLs deserve to exist at all.
Usually the answer is no, and the cleanest fix is at the source: stop generating links to empty calendar months, keep filter combinations behind controls a crawler does not follow, and drop session identifiers out of addresses. Where a URL must exist for people but not for search, use a robots meta tag rather than a block, so the page can still be read and its links still count. On a large shop this is ordinary technical SEO housekeeping, and it usually starts with the rules behind faceted navigation.
