How PerplexityBot works
PerplexityBot is the crawler Perplexity uses to find and index pages its answer engine can retrieve and cite. It carries its own user-agent token and reads robots.txt, and Perplexity publishes both the token names and the addresses it crawls from. A separately named agent handles fetches triggered by a user’s own request rather than by a scheduled crawl.
What makes it different from a training crawler is the output. Perplexity’s answers carry numbered source links, so a page it has retrieved can appear as a visible, clickable citation rather than dissolving into an unattributed summary. Access to your site and appearance in the answer are tightly connected here in a way they are not elsewhere.
Why PerplexityBot matters
Being cited is the closest thing an answer engine offers to a ranking. The link sits inside the answer, a reader who wants the detail follows it, and you receive a visit from somebody who has already read a summary of your position. Blocking the crawler removes that possibility outright.
It also gives you something you can actually measure. Unlike an assistant that paraphrases without linking, a cited answer can send referral traffic that appears in analytics, so you can judge whether the visibility turns into enquiries instead of guessing at it.
Common mistakes with PerplexityBot
The first is trusting the name in the log line. A user-agent string is written by whoever sends the request, and scrapers routinely borrow well-known crawler names. If a crawler’s identity matters to you — for billing, for licensing, or for a block you intend to enforce — verify the requesting address against the ranges the operator publishes. Anything concluded from the string alone is a guess.
The second is blocking on reputation. Answer engines attract argument, and blanket bans are often copied from a forum thread rather than reasoned from a site’s own logs. Look at what the crawler costs you and what the citations bring you before deciding.
The third is expecting robots.txt to govern everything. It is a convention aimed at crawling, and a fetch made because a user pasted your link is a different kind of request. Anything you genuinely need withheld belongs behind authentication, not behind a Disallow line.
What to do about it
Decide on evidence. Pull the user-agent tokens out of your server logs, see how often they appear and what they cost you, then check analytics for referrals from the answer engine. Those two readings, taken from your own site, are worth more than any general advice about whether to allow it.
If you want the citations, make the pages easy to cite: a clear answer near the top, facts stated plainly, a visible date and a named author, because an engine has to be able to attribute a claim before it will repeat it. If you do not want them, write an explicit rule naming the tokens and put sensitive material behind a login. Either way, revisit the decision when your traffic mix changes rather than leaving it in place forever, and watch your wider AI visibility alongside it.