How OAI-SearchBot works
OAI-SearchBot is the crawler OpenAI uses to build the index behind ChatGPT’s search results. It fetches pages so they can be found, summarised and linked when someone asks a question, and OpenAI states that what it collects is not used to train models — that is GPTBot‘s job. It reads robots.txt under its own user-agent token and crawls from published address ranges.
Three kinds of visit are worth telling apart. A scheduled crawl by OAI-SearchBot keeps the index current. A live fetch happens when a user asks the assistant to open one specific page, and that arrives under a different token again. A training crawl is a fourth thing entirely. Rules written for one of them do not apply to the others.
Why OAI-SearchBot matters
This is the crawler that decides whether you can be quoted at all. If it cannot reach your pages, your business is not ranked lower in assistant answers — it is absent from them. For a consultancy or a local service, being the source an assistant names is worth more than a mid-page listing, because the mention arrives inside the answer rather than beside it.
It also changes the shape of your traffic. Visits that begin in an assistant tend to arrive already informed, with the comparison done elsewhere, so they are fewer and further along. In a monthly report that is easy to misread as falling interest.
Common mistakes with OAI-SearchBot
The most common is collateral damage. A rule written in robots.txt to keep AI crawlers away from your content usually names every AI user agent someone could find, this one included, and the owner has quietly opted out of being cited while still turning up on a competitor’s comparison.
The second is blocking at the edge rather than in the file. Firewalls and bot-protection services often treat unfamiliar crawlers as threats, so robots.txt says one thing and the server does another. Only your logs will show which is winning.
The third is reading a citation as a ranking. Being quoted once is not a position you hold; it depends on the question asked and on which pages were retrieved that day.
What to do about it
Allow it unless you have a specific reason not to, and make sure the allowance survives your firewall as well as your robots file. Then make the pages worth quoting: a direct answer near the top, facts written as text rather than pictures, and a plain statement of who you are and where you work, because an assistant has to be able to attribute a claim before it will repeat it.
Measure it honestly. Confirm access in your server logs, watch for assistant referrers in analytics, and put your customers’ own questions to the assistants, noting whether you are named and whether what is said about you is true. Fixing the page that produced a wrong answer is the only lever you genuinely hold over AI visibility.