SEO

Crawl Demand

Also called Crawl scheduling

How much a search engine actually wants to crawl a site's URLs, judged by popularity and how often they change.

Quick facts: Crawl Demand

Category
SEO
Also called
Crawl scheduling
Level
Advanced
Affects
How often pages are re-crawled, freshness in results, crawl budget
Where to see it
Google Search Console (Crawl stats, URL inspection), server access logs
In this article4
  1. What crawl demand measures
  2. Why crawl demand matters
  3. Where crawl demand goes wrong
  4. What to do about it

What crawl demand measures

Crawling has two halves. One is how much your server can take, which shows up as crawl rate. The other is how much the search engine actually wants your pages, and that is crawl demand. A site can have plenty of spare capacity and still be crawled rarely, simply because nothing about it suggests there is anything new worth collecting.

Demand is built from a few judgements the engine makes on its own. How popular a URL appears to be, judged from links pointing at it and from interest in it. How often the content has genuinely changed in the past, because a page that never changes does not need re-reading. How important the URL looks within your own structure, which is why pages linked from the homepage and main navigation get visited more than pages buried at the end of a pagination chain. And whether something site-wide has happened, such as a migration or a change of address, which temporarily raises demand across everything.

Why crawl demand matters

Because for most sites this, and not server capacity, is the real constraint. Owners of large catalogues often assume Google is being held back by hosting. Far more often Google has simply decided that most of those URLs are not worth returning to, and the fix is editorial rather than technical.

It also explains the pattern almost every business notices eventually: important pages get re-crawled quickly while older, deeper pages sit untouched for long stretches. That is demand doing its job. If a page you care about is in the slow group, something about how it is linked, how it changes and who visits it is telling the engine it does not matter.

Where crawl demand goes wrong

Mass-producing URLs is the usual way to destroy it. Filter and sort combinations, session parameters, internal search results, near-identical location or service pages spun from a template. Each one is a URL the engine may have to consider, and a site full of pages that say almost nothing teaches the engine that the site is not worth checking often.

Faking freshness is the second. Bulk-updating dates, republishing old posts with no real change, or re-saving every page to bump the last-modified stamp all get discounted quickly, because the engine compares what it fetched with what it had. What survives that comparison is content that actually changed.

The third is orphaning. A page nothing links to and nobody visits has almost nothing to generate demand from, however good it is.

What to do about it

Start by cutting the URLs that add nothing. Consolidate near-duplicates, stop generating parameter variants that nobody would ever want indexed, and keep thin archives out of the index. Reducing index bloat concentrates attention on the pages that earn it.

Then feed the pages you care about. Link them from pages that already get crawled often, mention them in new content, and give them a route from the navigation rather than only from a sitemap. Update pages when there is something real to update, and leave the rest alone.

Finally, judge results by what gets re-crawled, not by how much. In the crawl stats report, watch whether requests are going to money pages or to noise. Shifting that mix is the point of the exercise, and it is usually the most valuable part of a technical SEO project on a large site.

Do and do not

Do

  • Consolidate or remove URLs that add nothing
  • Link important pages from pages crawled often
  • Update pages that genuinely need updating

Do not

  • Fake freshness by touching dates without changes
  • Publish thousands of near-identical template URLs
  • Assume more crawling means better rankings

Questions people ask about this

How is crawl demand different from crawl budget?

Crawl budget is the outcome of two things working together: what your server can handle, and how much the search engine wants your pages. Crawl demand is that second half, the appetite side. A site with fast hosting and low demand will still be crawled lightly, because capacity was never the limiting factor.

How do I increase crawl demand for an important page?

Give the engine reasons to come back. Link to it from pages that are crawled often, especially the homepage, main navigation and recent content. Earn external links and real visits to it. Update it when there is genuinely something to update. Remove competing near-duplicate pages so the value is not split between them.

Does publishing more often increase crawl demand?

It can, but only if the content is worth crawling. Regular publishing that people link to and read tends to raise interest in the site. Regular publishing of thin, similar pages does the opposite, because the engine learns that new URLs on your site rarely contain anything new. Quality decides which way it goes.

Related terms

Found this useful?

Share it, or ask an AI to summarise it

Back to the glossary

Knowing the term is the easy part

Applying it to your own site and budget is the work. Book a call and I will tell you what actually applies to you.