SEO

Crawl Stats Report

Also called Crawl stats

Search Console's summary of how often Googlebot requested files from your site, and what your server sent back.

Quick facts: Crawl Stats Report

Category
SEO
Also called
Crawl stats
Level
Advanced
Affects
Indexing speed, server load, crawl budget, technical diagnosis
Where to see it
Google Search Console (Settings → Crawl stats), server access logs
In this article4
  1. What the crawl stats report measures
  2. Why the crawl stats report matters
  3. Common mistakes with crawl stats
  4. How to act on it

What the crawl stats report measures

Every time Googlebot asks your server for a file — a page, an image, a stylesheet, a script — that request is recorded. The crawl stats report, tucked under Settings in Google Search Console, summarises those requests across the whole property for a recent rolling window. It gives you the number of requests, the total download size and the typical server response time, plotted as trends rather than single figures.

Below the chart, the same requests are grouped by response code, by file type, by purpose and by crawler. Purpose separates refresh crawls, where Google is re-checking a URL it already knows, from discovery crawls, where it is fetching something new. Crawler separates smartphone Googlebot from the desktop, image and ad crawlers, which is how you tell ordinary indexing traffic apart from a shopping feed check or an ad landing page review.

Why the crawl stats report matters

It is the one place in Search Console that shows your site from the server’s side rather than the index’s side. A page that never appears in search might have been crawled and rejected, or it might never have been requested at all, and those two problems have completely different fixes. Crawl stats tell you which one you are dealing with.

It is also an early warning system. Response times drifting upward, a rise in server errors, or a slump in total requests usually shows here before rankings move. On cheap shared hosting — common for smaller sites in Nepal — a slow response under crawl load is a genuine constraint, and this report is where it becomes visible. If your own server is throttling how much Google can fetch, no amount of content work will help until the hosting is fixed.

Common mistakes with crawl stats

The first is treating more crawling as better. A rising request count can mean Google has found a faceted navigation or an events calendar generating endless URLs, and attention is being spent on pages you never wanted in the index. Read request volume next to what is actually getting indexed, never on its own.

The second is reading it like a page-level tool. Crawl stats are aggregated for the whole property and will not tell you when one particular URL was last fetched — that is what the URL Inspection tool is for. The third is panicking at a single spike. Crawling is naturally uneven, and one unusual day in a chart is not a trend.

How to act on it

Start with host status, which flags whether robots.txt fetching, DNS or server connectivity failed recently. Then read the response codes. Server errors and not-found responses in any quantity deserve investigation, because both waste requests and both suggest something is broken.

If requests are being spent on the wrong URLs, the fix is structural: block genuinely useless paths in robots.txt, stop linking to them internally, and make sure your XML sitemap lists only pages you want indexed. If response time is the problem, it belongs with hosting and technical SEO work rather than with content. A small brochure site needs a glance at this report once or twice a year; a large catalogue deserves a monthly look alongside crawl budget.

Do and do not

Do

  • Check host status before reading any of the charts
  • Investigate server errors and not-found responses first
  • Compare request volume against what is actually indexed

Do not

  • Assume more crawling means better rankings
  • Use it to check when one URL was fetched
  • React to a single day's spike as a trend

Questions people ask about this

How often does the crawl stats report update?

It refreshes daily, with a lag of a day or two, so the most recent points on the chart are usually incomplete and dip for no real reason. Search Console also keeps only a recent rolling window of history rather than a full archive. If you need crawl history going back further, export it regularly or read your own server logs.

My crawl requests dropped suddenly. Is that a problem?

Sometimes. Google reduces crawling when a server responds slowly or returns errors, so check host status and response times first. It also crawls less when a site genuinely has less new or changed content, which is normal and harmless. A drop that lines up with a hosting change, a redesign or a robots.txt edit is worth investigating the same day.

Do I need the crawl stats report if my site is small?

Rarely. A site with a few dozen pages is usually crawled without difficulty, and the page indexing report will tell you far more about why something is missing from search. Crawl stats earn their place once you have thousands of URLs, a large product catalogue, or a server that has already proved unreliable.

Related terms

Found this useful?

Share it, or ask an AI to summarise it

Back to the glossary

Knowing the term is the easy part

Applying it to your own site and budget is the work. Book a call and I will tell you what actually applies to you.