SEO

Crawl-Delay

Also called Crawl delay directive

A robots.txt line asking a crawler to wait between requests; Bing respects it, Google does not.

Quick facts: Crawl-Delay

Category
SEO
Also called
Crawl delay directive
Level
Advanced
Affects
Server load, crawl speed, how quickly changes are found
Where to see it
robots.txt file, Bing Webmaster Tools, server access logs
In this article4
  1. How crawl-delay works
  2. Why crawl-delay matters
  3. Where crawl-delay goes wrong
  4. What to do about it

How crawl-delay works

The directive sits inside your robots.txt file, under a user-agent block, and names a waiting time in seconds that a crawler should leave between requests. A crawler that honours it queues its requests one after another instead of firing several at once, which lowers the load any single bot can put on a server.

Support is patchy, and that patchiness is really the whole story of this term. Crawl-delay was never part of the original robots exclusion standard, so each engine decided for itself whether to read it. Bing reads it and documents it. Googlebot does not: Google states plainly that it ignores the line and sets its own pace from how your server behaves. Plenty of smaller crawlers and scrapers ignore it too, and an aggressive scraper is usually the traffic you wanted to slow down in the first place.

Why crawl-delay matters

Mostly it matters as a warning sign. If somebody has put a crawl-delay in your robots.txt, then at some point someone decided bots were hurting the site. On shared hosting, still the normal starting point for small businesses in Nepal, a burst of crawler requests really can make pages slow for paying visitors, and throttling looks like the cheap fix.

The trade-off is that slowing a crawler slows discovery. New pages take longer to be found, changed pages take longer to be re-read, and a large site can end up with whole sections visited rarely. That cost is paid quietly and nobody attributes it to a single line in a text file, which is how a stale crawl-delay survives for years.

Where crawl-delay goes wrong

The most common error is expecting it to control Google. It does not, so anyone who adds it to stop Googlebot hammering the server has fixed nothing while slowing Bing for no reason at all.

The second is using it to paper over hosting. If one crawler at a normal pace can hurt the site, ordinary traffic will hurt it too, whether that is a campaign launch, a news mention or a busy evening. Throttling bots hides the symptom and leaves real visitors with the same slow pages.

The third is a generous delay on a big site. Apply a long wait to every URL and the crawler simply cannot get round the site in any useful time, which quietly starves crawl budget on exactly the sites that need it most.

What to do about it

Start with the server logs and establish which bot is actually responsible. Scrapers routinely claim to be Googlebot, and verifying that requests come from Google’s own address ranges settles the argument in minutes.

If the load is real and it genuinely is Googlebot, the supported signal is an HTTP response rather than a robots.txt line: returning a temporary service unavailable or too many requests status for a short period tells Google to back off, and it will. Keep sending those codes for a long stretch and Google may begin dropping the URLs, so treat it as an emergency brake and nothing more. For Bing, set the rate inside Bing Webmaster Tools where you can see what you have chosen. Then deal with the underlying cause. Caching, a lighter theme, fewer database queries or better hosting will do more for both bots and buyers than any directive, and that work belongs in a proper technical SEO pass.

Do and do not

Do

  • Fix slow server responses before throttling any crawler
  • Control Bingbot inside Bing Webmaster Tools instead
  • Check server logs to confirm which bot is the load

Do not

  • Expect Googlebot to obey a crawl-delay line
  • Throttle crawlers to hide a hosting problem
  • Leave a delay in place that nobody ever reviews

Questions people ask about this

Does Google support crawl-delay?

No. Google documents that Googlebot ignores the crawl-delay directive in robots.txt and works out its own pace from how quickly and reliably your server responds. If Google's crawling is genuinely overloading the site, the supported signals are temporary service-unavailable or too-many-requests responses, plus fixing the server capacity causing the problem.

Will a crawl-delay hurt my rankings?

Not directly, because it is not a ranking factor. The harm is indirect. Crawlers that do respect the delay visit fewer pages in the same time, so new and updated content is discovered more slowly. On a small brochure site that barely matters. On a large site with changing stock, prices or listings, sections can end up rarely refreshed.

How do I slow down a crawler that ignores robots.txt?

Robots.txt is a request, not a lock, so a badly behaved bot needs a server-side answer. Rate limiting at the web server or firewall, a rule at your CDN, or blocking the offending user agent and address range all work. Verify first that the bot is not a search engine you want, because blocking Googlebot removes you from search.

Related terms

Found this useful?

Share it, or ask an AI to summarise it

Back to the glossary

Knowing the term is the easy part

Applying it to your own site and budget is the work. Book a call and I will tell you what actually applies to you.