SEO

Meta Robots Tag

Also called Robots meta tag, meta name robots

A tag in a page's head telling search engines whether they may index the page and follow its links.

Quick facts: Meta Robots Tag

Category
SEO
Also called
Robots meta tag, meta name robots
Level
Intermediate
Affects
Indexing, which pages appear in search, crawl efficiency
Where to see it
Google Search Console URL Inspection and Page indexing report, Screaming Frog (Directives tab), browser view source
In this article4
  1. How the meta robots tag works
  2. Why the meta robots tag matters
  3. Where the meta robots tag goes wrong
  4. How to act on it

How the meta robots tag works

The tag sits in the head of an HTML page and carries instructions for search engine crawlers. The two values people meet first are the indexing pair — index or noindex, deciding whether the page may appear in results — and the link pair, follow or nofollow, deciding whether the links on that page are followed onward. If no tag is present, crawlers assume they may index the page and follow its links, which is why most pages need no tag at all.

Other values control how the page is displayed rather than whether it is indexed. Noarchive suppresses a cached copy, nosnippet stops a text preview being shown, and noimageindex keeps images on the page out of image results. You can also address one crawler specifically by naming it in place of the general robots value, which is how a page is hidden from one search engine while staying visible in another.

Why the meta robots tag matters

It is the only reliable way to keep a page out of search results while leaving it working for people. Thank-you pages, internal search results, filtered listings, staging copies, print versions and pages that exist purely for a campaign all belong in that category: useful to visitors, unhelpful in an index.

It also protects the rest of the site. A large volume of near-identical pages competing with each other makes it harder for a search engine to tell which one deserves to rank. Removing the noise with a directive is usually faster and safer than deleting pages people may still be using.

Where the meta robots tag goes wrong

The classic failure is combining it with a block in robots.txt. A crawler that is not allowed to fetch the page never sees the tag, so the instruction is never read. The page can still be listed from links pointing at it, with no description, and the more you tighten the block the longer it stays that way. If you want a page removed, it has to remain crawlable.

The expensive failure is a site-wide noindex left switched on after a launch or a redesign. It is set deliberately while a staging site is being built, then survives the move to the live domain, and traffic disappears for reasons nobody can find. The third failure is subtler: using noindex as a way of hiding weak pages rather than improving or removing them, which leaves the underlying content problem untouched.

How to act on it

Check templates rather than individual pages. Directives are almost always applied by a theme or a plugin across a whole type of page, so one product page tells you what every product page is doing. A crawl that reports directives site-wide will show a stray noindex in seconds.

Make the check part of every launch, alongside the other tasks that only matter once. Confirm the live domain is indexable, spot-check a page from each template with URL Inspection in Google Search Console, and remember that removal is not instant — the page has to be recrawled before the instruction takes effect. Keeping this tidy is routine technical SEO maintenance.

Do and do not

Do

  • Check directives per template, not per page
  • Leave a page crawlable if you want noindex obeyed
  • Verify indexability on the live domain after every launch

Do not

  • Block a page in robots.txt and expect noindex to work
  • Use noindex instead of fixing or removing weak pages
  • Assume a removal takes effect before the page is recrawled

Questions people ask about this

What is the difference between the meta robots tag and robots.txt?

Robots.txt controls whether a crawler may fetch a page at all. The meta robots tag controls what happens after it has been fetched, such as whether the page may be indexed. They are easily confused because blocking a page in robots.txt prevents the tag from ever being read, so the instruction inside it is lost.

Will noindex remove a page from Google immediately?

No. The page has to be crawled again before the instruction is seen, and how quickly that happens depends on how often the page is visited. Important pages are recrawled sooner than forgotten ones. If you need something gone urgently, use the removals tool in Search Console alongside the tag rather than instead of it.

Do I need a meta robots tag on every page?

No. With no tag present, crawlers assume a page may be indexed and its links followed, which is what you want for almost everything. Add a tag only where you need different behaviour. Adding an explicit index and follow tag everywhere does no harm but achieves nothing either.

Related terms

Found this useful?

Share it, or ask an AI to summarise it

Back to the glossary

Knowing the term is the easy part

Applying it to your own site and budget is the work. Book a call and I will tell you what actually applies to you.