SEO

Allow

Also called robots.txt allow directive

The robots.txt rule that reopens one path inside an area a Disallow rule has already closed off.

Quick facts: Allow

Category
SEO
Also called
robots.txt allow directive
Level
Intermediate
Affects
Which blocked paths stay crawlable, page rendering, crawl budget
Where to see it
Your robots.txt file, Google Search Console robots tester
In this article4
  1. How Allow works
  2. Why Allow matters
  3. Common mistakes with Allow
  4. How to act on it

How Allow works

Allow is a rule in robots.txt that reopens a path inside an area you have otherwise blocked. On its own it does nothing useful, because everything is crawlable by default; its only job is to override a Disallow rule that is drawn wider than you intended.

The interesting part is how a conflict is settled when both rules match the same address. Google resolves it by specificity: the rule with the longer matching path wins, and where two rules match equally, the less restrictive one — the Allow — takes precedence. That means you do not fix an over-broad block by moving lines around. You fix it by writing a more specific exception.

One caution. Allow was not part of the original robots standard; the major search engines added support for it and it now sits in the current specification, but smaller crawlers and older tools may honour only the blocks. Anything you rely on this rule for should be verified against the crawler you actually care about.

Why Allow matters

Blocking works on path prefixes, so a single sensible-looking rule often catches more than it should. An administrative folder that must stay closed may contain the one script the front end calls to load content. A resource directory you want kept out of crawling may hold the stylesheets and images the crawler needs to render your pages properly. Allow is what stops those cases forcing a choice between blocking too much and blocking nothing.

It matters most where rendering is at stake. If a crawler cannot fetch the files that build the page, it sees a broken version of your site and judges the content on that. A narrow exception inside a broad block is often the difference between a page being understood and a page being misread.

Common mistakes with Allow

The commonest is treating it as an instruction to crawl. It is not a request for attention and it does not affect indexing at all — it only removes a block. Adding it to a path that was never blocked achieves precisely nothing, and files full of such rules are usually a sign someone was guessing.

The second is assuming rule order decides the outcome. It does not for Google, where specificity decides, and it may for other crawlers that take the first match they find. A robots file that only works because of the sequence of its lines is fragile.

The third is patching rather than fixing. If a block needs several exceptions to be workable, the block itself is drawn wrong, and rewriting it is cleaner than maintaining a growing list of holes.

How to act on it

Use it in one situation only: when a block you genuinely want is catching something you genuinely need crawled. Write the exception as specifically as you can, then test the exact address in Search Console’s robots tester rather than assuming the rule reads the way you meant it.

Recheck after any platform update, since plugins and themes move file paths without warning. And keep the whole file short — a robots file you can read in one glance is one where an over-broad block, and the crawl budget it wastes, is obvious rather than buried.

Do and do not

Do

  • Use it only to reopen something a real block caught
  • Write the exception as specifically as the path allows
  • Test the exact address in Search Console

Do not

  • Add Allow rules for paths nothing blocks
  • Depend on the order of lines in the file
  • Keep patching a block that needs rewriting

Questions people ask about this

Do I need Allow rules if I am not blocking anything?

No. Everything on a site is crawlable unless a rule says otherwise, so an Allow rule on an unblocked path does nothing at all. It only has an effect inside an area a Disallow rule already covers. A robots file full of Allow lines with no matching blocks is clutter, and it can hide the rules that do matter.

Does Allow override Disallow?

For Google, whichever rule matches the address more specifically wins, and if two rules match equally the less restrictive one applies, which means Allow. Line order does not decide it. Other crawlers may behave differently, some taking the first matching rule, so avoid writing a file whose meaning depends on the sequence of its lines.

Will an Allow rule get a page indexed?

No. It only removes a crawl block, and being crawlable is not the same as being indexed. Whether a page appears in search still depends on its content, its links and any indexing instructions on the page itself. If a page you want in search is missing, robots.txt is only one of the things to check.

Related terms

Found this useful?

Share it, or ask an AI to summarise it

Back to the glossary

Knowing the term is the easy part

Applying it to your own site and budget is the work. Book a call and I will tell you what actually applies to you.