SEO

Crawl Rate

Also called Crawl capacity limit

How many requests a search engine makes to a site over a period, and how fast it is willing to go.

Quick facts: Crawl Rate

Category
SEO
Also called
Crawl capacity limit
Level
Advanced
Affects
How quickly changes are noticed, server load, crawl budget
Where to see it
Google Search Console (Crawl stats), server access logs, Bing Webmaster Tools
In this article4
  1. What crawl rate measures
  2. Why crawl rate matters
  3. Where crawl rate goes wrong
  4. What to do about it

What crawl rate measures

Crawl rate is the speed side of crawling: how many requests a search engine sends to your server in a given stretch of time, and how many it is willing to run at once. It says nothing about whether the pages fetched are any good. It is a measure of traffic between two machines.

Google works out its own ceiling and adjusts it constantly. The main input is how your server behaves. Fast, reliable responses tell Googlebot there is room to fetch more, so it raises the rate. Slow responses, timeouts and server errors tell it the site is struggling, so it backs off, sometimes sharply and for longer than owners expect. Because the adjustment is automatic, crawl rate is really a report on server health as much as on search behaviour.

Speed is only half the picture, though. The engine also has to want the pages, and that appetite is crawl demand. Capacity and demand together are what people mean when they talk about crawl budget.

Why crawl rate matters

It matters in two situations, and in neither of them is a bigger number the goal.

The first is a site under strain. Where hosting is modest, and shared hosting is still the norm for small businesses in Nepal, crawler traffic competes with customers for the same limited resources. If pages slow down when bots are busy, that is a hosting problem showing itself through crawl rate.

The second is a site that changes constantly. A shop with shifting stock, a listings site or a large catalogue needs its pages re-fetched often enough that search results still match reality. If the rate collapses, results go stale, and people click through to items that have gone.

Where crawl rate goes wrong

The most common misreading is treating a rising crawl rate as good news. It usually means more URLs exist, not that the site has improved, and a jump often traces back to filters, search pages or session parameters generating endless addresses that were never worth crawling.

The opposite mistake is ignoring a collapse. A sudden, sustained drop in daily requests is one of the clearest early signals of a technical fault: a robots.txt rule deployed by accident, a firewall now blocking the crawler, a server timing out, a migration that broke internal links.

The third is fighting the symptom. Throttling crawlers to keep a slow site upright hides a hosting problem that is already costing you visitors, because a server too slow for a crawler is too slow for a customer on a phone.

What to do about it

Read the crawl stats report in Search Console rather than guessing, and read total requests next to average response time. Rising requests with steady response times is fine. Rising requests with rising response times means the server is being pushed past what it can comfortably do.

Then look at what is being fetched. If most requests go to images, scripts or parameter URLs rather than to pages you want indexed, the fix is on your side: caching, compression, tidier URL handling and fewer pointless addresses. Clear the server errors first, since they suppress the rate on their own, and treat a healthy, stable rate as the by-product of a fast site rather than as a target to chase in its own right. That is ordinary technical SEO work, and it pays for itself in page speed too.

Do and do not

Do

  • Read total requests next to average response time
  • Clear server errors before worrying about crawl volume
  • Check which file types are consuming the requests

Do not

  • Read a rising crawl rate as a ranking signal
  • Throttle Googlebot to mask a hosting problem
  • Ignore a sudden collapse in daily requests

Questions people ask about this

Can I control how fast Google crawls my site?

Not directly any more. Google sets the rate itself from how your server responds. If crawling is genuinely overloading the site you can signal that with temporary error responses, but the supported answer is to make the server respond faster and more reliably, at which point the rate settles at a level the site can handle.

Is a higher crawl rate better for rankings?

No. Crawl rate is not a ranking signal. It describes how much of your site is being fetched, not how well it performs in results. A large increase is often a warning that unwanted URLs are being generated somewhere, and a healthy site can sit at a modest, steady rate for years without any problem.

My crawl rate dropped suddenly. What should I check?

Start with the things that stop a crawler outright: a changed robots.txt, a firewall or security plugin blocking the crawler, server errors, and slow response times. Then check whether pages were removed, redirected or noindexed in bulk. A sustained drop with no change on your side is unusual, so assume something on the site changed.

Related terms

Found this useful?

Share it, or ask an AI to summarise it

Back to the glossary

Knowing the term is the easy part

Applying it to your own site and budget is the work. Book a call and I will tell you what actually applies to you.