How crawling works
A search engine keeps a running list of addresses it intends to visit. A bot — Googlebot, in Google’s case — requests each one much as a browser would, reads the HTML that comes back, notes every link inside it and adds anything unfamiliar to the list. That loop is crawling, and it never stops.
Addresses reach the list in a few ways: links from pages already known, entries in an XML sitemap, redirects from addresses already known, and links from other websites. Before fetching anything from a site, a crawler reads its robots.txt file to see which paths it is allowed to request. Where a page assembles its content with JavaScript, the crawler has to render the page before there is anything to read, which is a second and slower step in the same process.
Why crawling matters
It is the first gate, and it is absolute. A page that is never fetched cannot be indexed, cannot rank and cannot be quoted in an answer box. Every piece of advice about writing better content quietly assumes the page was reachable in the first place.
It is also where a surprising number of invisible problems live. A product page reachable only through an on-site search box. A menu that assembles itself after the page has loaded. A page nobody ever linked to because it was built for one email campaign. None of these looks broken to a visitor handed the address, and all of them can be missing from search entirely.
Common mistakes with crawling
The classic launch error is a robots.txt file copied across from staging that tells every crawler to stay out of the whole site. It is a two-line file and it can hide an entire business. Read it on launch day, and read it again a week later.
Confusing crawled with indexed comes next: being fetched is permission to be considered, not a promise of inclusion. Blocking CSS and JavaScript is another, because the crawler then renders a version of the page your visitors never see and judges that instead. And leaning on a sitemap alone, with no links pointing at a page, may get it discovered but says nothing about whether it matters.
How to act on it
Use URL Inspection in Search Console on a page that matters. It reports whether Google has fetched the address, when it last did, what it saw and whether anything blocked it. Run it on one page from each template — home, service, article, product — rather than trying to check everything.
Then make the site easy to walk. Any page worth ranking should be reachable by following ordinary links from the home page in a small number of clicks. Keep robots.txt short and reviewed. Watch for server errors and slow responses, which quietly reduce how much gets fetched. Publish a sitemap listing the pages you want found, but treat it as a supplement to internal linking rather than a replacement — a page with nothing linking to it is an orphan whatever the sitemap claims. Keeping that plumbing sound is everyday technical SEO.