How robots.txt works
Robots.txt is a plain text file that lives at the root of a domain and nowhere else — a crawler looks for it at the top level of the host before requesting anything else. Inside, rules are grouped by user agent: a line naming the crawler, then Disallow lines listing path patterns it should not request, and Allow lines carving exceptions back out. A sitemap line can point crawlers at your list of addresses.
Two things about it surprise people. It governs crawling only — whether a bot may fetch an address — and says nothing about whether that address may appear in results. And it is voluntary. The major search engines honour it; scrapers and bad actors ignore it, because the file is a public request rather than a lock.
Why robots.txt matters
On a small brochure site it rarely changes anything, and the safest file is often an almost empty one. It earns its keep on large or untidy sites, where crawlers can spend their time on endless address combinations: faceted product filters, internal search results, session parameters, calendar pages stretching into the future. Steering bots away from those areas leaves more attention for pages that matter.
It also protects fragile parts of a site. Some servers strain when a crawler repeatedly hits a heavy search or export endpoint, and a disallow rule is the quickest way to stop that. Because the file is public, it is worth remembering that anyone can read it — including people looking for the admin paths you helpfully listed.
Common mistakes with robots.txt
The famous one is a single blocking line left over from a staging environment and pushed live with the rest of the site. It shuts out everything, and because the pages are never fetched, nothing on them can correct the mistake.
Almost as common is expecting the file to remove a page from search. Blocking prevents crawling, not indexing; a blocked address that other pages link to can still be listed, with no description, because the crawler was never allowed to look inside. Worse, if that page carries a noindex directive, the block guarantees the directive is never read. Pick one tool or the other, never both on the same URL.
The third is blocking stylesheet, script or image folders to save crawl effort. Search engines render pages in order to judge them; deny them the files that build the layout and they see something broken.
How to act on it
Keep the file short, deliberate and reviewed. Every line should exist for a reason someone can explain, because a stale rule is invisible until it costs you a whole section of the site. Write it yourself rather than accepting whatever a plugin generates by default.
Test before and after any change. Search Console reports the version Google last fetched and will tell you whether a specific address is allowed, which settles arguments quickly. Put the file on your launch checklist too — it is one of the first things to verify when a site moves from staging to live, alongside the rest of a technical SEO handover.
