SEO

429

Also called Too Many Requests, rate limiting

The response a server gives when a client has made too many requests in too short a time.

Quick facts: 429

Category
SEO
Also called
Too Many Requests, rate limiting
Level
Advanced
Affects
Crawl rate, indexing speed, audit accuracy
Where to see it
Search Console (Crawl stats, host status), server and CDN logs, Screaming Frog
In this article4
  1. How a 429 works
  2. Why a 429 matters
  3. Common mistakes with 429s
  4. How to act on it

How a 429 works

A server, a firewall or a content delivery network counts how many requests arrive from the same source in a period of time. When that count passes a threshold, further requests are turned away with a 429 until the pressure drops. The response can carry a header telling the client how long to wait before trying again, and well-behaved clients honour it.

Unlike a refusal on grounds of permission, this is temporary and about volume alone. The content exists and the visitor is welcome; they have simply asked too often. Limits are commonly set at the hosting plan, in a security plugin, at the CDN, or by an application interface protecting its own capacity.

Why a 429 matters

Search engine crawlers treat rate limiting as a request to slow down, and they do slow down. That is the correct behaviour and it is also the problem: if the limit is too tight, crawling drops away, new pages take longer to be discovered and changes take longer to be noticed. Sustained rate limiting can leave a site being visited far less often than its publishing schedule needs.

It also quietly corrupts your own analysis. Crawls from tools such as Screaming Frog come back with pages missing, and it is easy to read those gaps as broken pages when the site was simply refusing to answer that quickly. Shared networks make this worse — an office, a campus or a mobile carrier can put many people behind a single address, which is common in Nepal, so a limit written per address can lock out a whole building at once.

Common mistakes with 429s

The most frequent is setting one aggressive limit for every client and never revisiting it. A default meant to stop abusive scraping will happily throttle a search engine as well, and the cost of that is invisible until indexing slows.

The second is crawling your own site at full speed and then reporting the resulting gaps as site errors. The third is using rate limiting as a substitute for capacity: if normal human traffic triggers the limit, the answer is more capacity or better caching, not a tighter threshold. And a 429 used where a permanent refusal is meant sends the wrong message entirely, because it invites the client to come back and try again.

How to act on it

Find the layer doing the counting before touching anything, since the same site can have limits at the host, the firewall and the CDN at once. Check the crawl stats report in Search Console for host status problems, and read your own crawler’s error column rather than assuming a clean run.

Give verified search engine crawlers a looser limit than anonymous traffic, and slow your own audits down with a crawl delay so your tooling stops competing with real visitors. If the limit is being hit during ordinary business hours, the hosting plan or the caching setup is undersized and no threshold will fix that. Persistent rate limiting on a growing site is a hosting decision as much as a technical SEO one.

Do and do not

Do

  • Give verified search engine crawlers a looser limit
  • Slow your own audit crawls with a delay
  • Check crawl stats for repeated host status problems

Do not

  • Apply one aggressive limit to every client
  • Read rate-limited pages in a crawl as broken
  • Use limits to cover for undersized hosting

Questions people ask about this

Does a 429 harm my search rankings?

Not as a penalty, but it slows discovery. Crawlers respond to rate limiting by visiting less often, so new pages take longer to appear and edits take longer to register. An occasional limit during a traffic spike is harmless. Limiting that happens every day means the crawler is being held back permanently, and that does cost you.

My SEO tool keeps getting 429 errors. What should I do?

Slow the crawl down. Most tools let you set requests per second and a delay between them, and a gentler setting will usually complete where a fast one fails. If you own the site, you can also allow your own address through the limit temporarily. Do not read the missing pages as broken until the crawl completes cleanly.

Is rate limiting the same as blocking?

No. Rate limiting is temporary and about volume: the client is welcome, it has simply asked too often, and it should try again shortly. Blocking is a refusal on principle and does not invite a retry. Using the wrong one confuses clients, because a temporary limit tells them to come back while a refusal tells them not to bother.

Related terms

Found this useful?

Share it, or ask an AI to summarise it

Back to the glossary

Knowing the term is the easy part

Applying it to your own site and budget is the work. Book a call and I will tell you what actually applies to you.