SEO

Robots.txt

Also called robots exclusion protocol

A plain text file at a domain's root that tells crawlers which paths they may request.

Quick facts: Robots.txt

Category
SEO
Also called
robots exclusion protocol
Level
Intermediate
Affects
Crawl efficiency, server load, what search engines can render
Where to see it
yourdomain.com/robots.txt, Search Console, Screaming Frog
In this article4
  1. How robots.txt works
  2. Why robots.txt matters
  3. Common mistakes with robots.txt
  4. How to act on it

How robots.txt works

Robots.txt is a plain text file that lives at the root of a domain and nowhere else — a crawler looks for it at the top level of the host before requesting anything else. Inside, rules are grouped by user agent: a line naming the crawler, then Disallow lines listing path patterns it should not request, and Allow lines carving exceptions back out. A sitemap line can point crawlers at your list of addresses.

Two things about it surprise people. It governs crawling only — whether a bot may fetch an address — and says nothing about whether that address may appear in results. And it is voluntary. The major search engines honour it; scrapers and bad actors ignore it, because the file is a public request rather than a lock.

Why robots.txt matters

On a small brochure site it rarely changes anything, and the safest file is often an almost empty one. It earns its keep on large or untidy sites, where crawlers can spend their time on endless address combinations: faceted product filters, internal search results, session parameters, calendar pages stretching into the future. Steering bots away from those areas leaves more attention for pages that matter.

It also protects fragile parts of a site. Some servers strain when a crawler repeatedly hits a heavy search or export endpoint, and a disallow rule is the quickest way to stop that. Because the file is public, it is worth remembering that anyone can read it — including people looking for the admin paths you helpfully listed.

Common mistakes with robots.txt

The famous one is a single blocking line left over from a staging environment and pushed live with the rest of the site. It shuts out everything, and because the pages are never fetched, nothing on them can correct the mistake.

Almost as common is expecting the file to remove a page from search. Blocking prevents crawling, not indexing; a blocked address that other pages link to can still be listed, with no description, because the crawler was never allowed to look inside. Worse, if that page carries a noindex directive, the block guarantees the directive is never read. Pick one tool or the other, never both on the same URL.

The third is blocking stylesheet, script or image folders to save crawl effort. Search engines render pages in order to judge them; deny them the files that build the layout and they see something broken.

How to act on it

Keep the file short, deliberate and reviewed. Every line should exist for a reason someone can explain, because a stale rule is invisible until it costs you a whole section of the site. Write it yourself rather than accepting whatever a plugin generates by default.

Test before and after any change. Search Console reports the version Google last fetched and will tell you whether a specific address is allowed, which settles arguments quickly. Put the file on your launch checklist too — it is one of the first things to verify when a site moves from staging to live, alongside the rest of a technical SEO handover.

Do and do not

Do

  • Keep the file short and justify every line
  • Test a changed file before and after deploying
  • Check it first when a site moves live

Do not

  • Use it to hide pages from search results
  • Block stylesheet, script or image folders
  • List private admin paths in a public file

Questions people ask about this

Does robots.txt stop a page appearing in Google?

No. It stops the page being fetched, which is a different thing. A blocked address that other websites link to can still be listed in results, usually with no description underneath because the crawler was never allowed to read it. To keep a page out of results, let it be crawled and give it a noindex directive instead.

Where should the robots.txt file live?

At the root of the host, so that it answers at yourdomain.com/robots.txt. Crawlers look there and nowhere else, so a copy in a subfolder is ignored. Each host counts separately, which means a subdomain needs its own file. If your site runs on WordPress and nobody has created one, the platform serves a basic virtual version.

Should I block my admin or login pages in robots.txt?

Think twice. The file is public, so listing a sensitive path advertises it to anyone curious enough to look. Most admin areas already sit behind a login and are not indexed anyway. If something genuinely must stay private, protect it with authentication; robots.txt is a polite request that well-behaved crawlers follow and everyone else ignores.

Related terms

Found this useful?

Share it, or ask an AI to summarise it

Back to the glossary

Knowing the term is the easy part

Applying it to your own site and budget is the work. Book a call and I will tell you what actually applies to you.