SEO

GPTBot

Also called OpenAI training crawler

OpenAI's crawler for gathering public web content that may be used to train its models; controlled through robots.txt.

Quick facts: GPTBot

Category
SEO
Also called
OpenAI training crawler
Level
Intermediate
Affects
Whether your content trains OpenAI models, crawl load, AI visibility decisions
Where to see it
robots.txt, server logs, OpenAI's published crawler documentation
In this article4
  1. How GPTBot works
  2. Why GPTBot matters
  3. Common mistakes with GPTBot
  4. What to do about it

How GPTBot works

GPTBot is OpenAI’s crawler for collecting publicly available web content that may be used to train its models. It identifies itself with the token GPTBot in the user-agent string, OpenAI publishes the address ranges it crawls from, and it reads robots.txt in the ordinary way. Disallow it and it stops requesting your pages.

The detail that matters is that it is one of several. OpenAI runs separate crawlers for separate jobs: OAI-SearchBot fetches pages so they can appear and be cited in ChatGPT’s search results, and a further agent fetches a single page when a user asks ChatGPT to open a link. Each carries its own token, so a rule aimed at GPTBot touches training collection and nothing else.

Why GPTBot matters

It puts a decision in front of ordinary businesses that used to belong to publishers: whether your writing may be used to train a commercial model. There is no payment, no attribution, and nothing you can point to afterwards to prove your work was included. Some owners object on principle. Others treat it as the price of being on the open web. Both positions are defensible; the drifting one is not.

It matters just as much because the decision is usually made by accident. Hosting companies and security plugins increasingly ship AI-crawler blocking switched on by default, and most site owners find out long afterwards, if ever.

Common mistakes with GPTBot

The costly mistake is blocking GPTBot in the belief that it removes you from ChatGPT. It does not. Retrieval and citation run through a different crawler, so a business that blocks the wrong one carries on supplying training data while vanishing from the answers where a customer might have found it — or blocks everything and then wonders why it is never mentioned.

The second is expecting a block to reach backwards. Disallowing a crawler stops future collection. It does not remove anything from a model that has already been trained.

The third is trusting the label. A user-agent string is self-reported, so any piece of software can claim to be GPTBot. If it matters to you, check the requesting address against OpenAI’s published ranges rather than the name in the log line.

What to do about it

Answer the two questions separately: do you want your content used for training, and do you want to be findable inside AI assistants. Different crawlers, different answers, written as explicit user-agent groups. Record the reasoning somewhere, because in a year nobody will remember why the rule is there.

Then verify instead of assuming. Read your robots.txt as it is actually served rather than as you remember writing it, look in your server logs for the tokens you care about, and check that your host or security plugin has not added a block you never asked for. The trade-off in blocking crawlers deserves one deliberate decision rather than an inherited default.

Do and do not

Do

  • Decide training and retrieval crawlers separately
  • Write an explicit robots.txt group for GPTBot
  • Check your host has not blocked it silently

Do not

  • Assume a block removes content already used
  • Block it hoping to leave ChatGPT search
  • Trust the user agent name without checking addresses

Questions people ask about this

Does blocking GPTBot remove my site from ChatGPT?

No. GPTBot collects content that may be used for training. Whether you appear and are cited in ChatGPT's search results depends on a different OpenAI crawler with its own name. Block only GPTBot and you can still be found and quoted; block everything and you lose the visibility along with the training. Decide the two separately.

How do I block GPTBot?

Add a user-agent group for GPTBot to the robots.txt file at the root of your domain and disallow the paths you want left alone. Crawlers follow the single most specific group that matches them and ignore the rest, so the rules must sit inside the GPTBot group. Do not assume it will also read your general rules.

Is blocking GPTBot bad for SEO?

No. It is a separate crawler from Googlebot and Bingbot, and it plays no part in search ranking or indexing. Blocking it affects only whether your content is collected for model training. The real risk is doing it clumsily: a broad rule that also catches search crawlers will cost you rankings very quickly.

Related terms

Found this useful?

Share it, or ask an AI to summarise it

Back to the glossary

Knowing the term is the easy part

Applying it to your own site and budget is the work. Book a call and I will tell you what actually applies to you.