How GPTBot works
GPTBot is OpenAI’s crawler for collecting publicly available web content that may be used to train its models. It identifies itself with the token GPTBot in the user-agent string, OpenAI publishes the address ranges it crawls from, and it reads robots.txt in the ordinary way. Disallow it and it stops requesting your pages.
The detail that matters is that it is one of several. OpenAI runs separate crawlers for separate jobs: OAI-SearchBot fetches pages so they can appear and be cited in ChatGPT’s search results, and a further agent fetches a single page when a user asks ChatGPT to open a link. Each carries its own token, so a rule aimed at GPTBot touches training collection and nothing else.
Why GPTBot matters
It puts a decision in front of ordinary businesses that used to belong to publishers: whether your writing may be used to train a commercial model. There is no payment, no attribution, and nothing you can point to afterwards to prove your work was included. Some owners object on principle. Others treat it as the price of being on the open web. Both positions are defensible; the drifting one is not.
It matters just as much because the decision is usually made by accident. Hosting companies and security plugins increasingly ship AI-crawler blocking switched on by default, and most site owners find out long afterwards, if ever.
Common mistakes with GPTBot
The costly mistake is blocking GPTBot in the belief that it removes you from ChatGPT. It does not. Retrieval and citation run through a different crawler, so a business that blocks the wrong one carries on supplying training data while vanishing from the answers where a customer might have found it — or blocks everything and then wonders why it is never mentioned.
The second is expecting a block to reach backwards. Disallowing a crawler stops future collection. It does not remove anything from a model that has already been trained.
The third is trusting the label. A user-agent string is self-reported, so any piece of software can claim to be GPTBot. If it matters to you, check the requesting address against OpenAI’s published ranges rather than the name in the log line.
What to do about it
Answer the two questions separately: do you want your content used for training, and do you want to be findable inside AI assistants. Different crawlers, different answers, written as explicit user-agent groups. Record the reasoning somewhere, because in a year nobody will remember why the rule is there.
Then verify instead of assuming. Read your robots.txt as it is actually served rather than as you remember writing it, look in your server logs for the tokens you care about, and check that your host or security plugin has not added a block you never asked for. The trade-off in blocking crawlers deserves one deliberate decision rather than an inherited default.
