How crawl-delay works
The directive sits inside your robots.txt file, under a user-agent block, and names a waiting time in seconds that a crawler should leave between requests. A crawler that honours it queues its requests one after another instead of firing several at once, which lowers the load any single bot can put on a server.
Support is patchy, and that patchiness is really the whole story of this term. Crawl-delay was never part of the original robots exclusion standard, so each engine decided for itself whether to read it. Bing reads it and documents it. Googlebot does not: Google states plainly that it ignores the line and sets its own pace from how your server behaves. Plenty of smaller crawlers and scrapers ignore it too, and an aggressive scraper is usually the traffic you wanted to slow down in the first place.
Why crawl-delay matters
Mostly it matters as a warning sign. If somebody has put a crawl-delay in your robots.txt, then at some point someone decided bots were hurting the site. On shared hosting, still the normal starting point for small businesses in Nepal, a burst of crawler requests really can make pages slow for paying visitors, and throttling looks like the cheap fix.
The trade-off is that slowing a crawler slows discovery. New pages take longer to be found, changed pages take longer to be re-read, and a large site can end up with whole sections visited rarely. That cost is paid quietly and nobody attributes it to a single line in a text file, which is how a stale crawl-delay survives for years.
Where crawl-delay goes wrong
The most common error is expecting it to control Google. It does not, so anyone who adds it to stop Googlebot hammering the server has fixed nothing while slowing Bing for no reason at all.
The second is using it to paper over hosting. If one crawler at a normal pace can hurt the site, ordinary traffic will hurt it too, whether that is a campaign launch, a news mention or a busy evening. Throttling bots hides the symptom and leaves real visitors with the same slow pages.
The third is a generous delay on a big site. Apply a long wait to every URL and the crawler simply cannot get round the site in any useful time, which quietly starves crawl budget on exactly the sites that need it most.
What to do about it
Start with the server logs and establish which bot is actually responsible. Scrapers routinely claim to be Googlebot, and verifying that requests come from Google’s own address ranges settles the argument in minutes.
If the load is real and it genuinely is Googlebot, the supported signal is an HTTP response rather than a robots.txt line: returning a temporary service unavailable or too many requests status for a short period tells Google to back off, and it will. Keep sending those codes for a long stretch and Google may begin dropping the URLs, so treat it as an emergency brake and nothing more. For Bing, set the rate inside Bing Webmaster Tools where you can see what you have chosen. Then deal with the underlying cause. Caching, a lighter theme, fewer database queries or better hosting will do more for both bots and buyers than any directive, and that work belongs in a proper technical SEO pass.
