How noindex works
Noindex is a directive carried in a page’s own response: either a meta robots tag in the HTML head, or an X-Robots-Tag in the HTTP headers, which is how non-HTML files such as PDFs are handled. A crawler fetches the page, reads the directive, and removes that URL from the index. The page still exists, still loads for visitors, and its links can still be followed — it simply stops being eligible to appear in results.
The detail that trips people up is that a crawler must be allowed to fetch the page before it can read the instruction. A URL disallowed in the site’s robots.txt file is never requested, so a noindex tag sitting inside it is never seen. Blocking a crawler and removing a page from the index are separate jobs, and doing both at once achieves neither.
Why noindex matters
Most websites produce far more URLs than anyone meant to publish: internal search results, tag and author archives, filtered product listings, thank-you pages, print versions, forgotten staging copies. Left in the index, these compete with the pages you actually want found, split signals across near-identical content, and spend crawler attention on addresses that will never earn a visit.
Noindex also protects pages that should exist but should not be discovered cold. A confirmation page is useful to the person who has just submitted a form and misleading to a stranger arriving from search — and every stranger who lands on it directly inflates your conversion counts with something that never happened.
Common mistakes with noindex
The costliest mistake is launching with the tag still switched on. WordPress carries a single setting that discourages search engines while a site is being built, and if it survives launch day every page ships with noindex and the site quietly stays invisible. Check it the hour the site goes live, not the week after.
The second is reaching for noindex when a canonical tag was the right tool. Two near-identical pages that both deserve to exist should point at one preferred version so their signals combine; noindex throws one of them away instead, links and all.
The third is treating it as privacy. A noindexed page is still public to anyone with the address, still readable, still shareable. If content must not be seen, it needs a password or a login, not a polite instruction to crawlers.
How to act on it
Decide indexability by template rather than page by page. Write down which types of URL on your site should be indexable — services, articles, locations, products — and which should not, then apply the rule in the template so new pages inherit the correct behaviour automatically. Page-level decisions taken one at a time drift within months.
Then verify rather than assume. Search Console’s URL Inspection tool reports what Google actually saw on the live address, which is the only account that counts, and a full crawl will surface any template that has picked up the directive by accident. Recheck after every migration, redesign or plugin change — this is routine work in any technical SEO review.
