Perplexity Crawler Checker
Can Perplexity find and cite your website? Test PerplexityBot (Perplexity's search index) and Perplexity-User (pages it opens when someone asks) live, including the firewall blocks that most often hide sites from Perplexity.
Check your home page or any page URL. Free, no sign-up, and we don't save results.
- Perplexity's crawlers first
- Live robots.txt check
- Firewall test
- Fix-it advice
Perplexity: every crawler at a glance
| User agent | What it does | Follows robots.txt | If you block it |
|---|---|---|---|
| PerplexityBot | Indexes sites so Perplexity can surface and link to them in its answers. Not used for model training. | Yes | Perplexity is far less likely to cite or link to you. |
| Perplexity-User | Fetches a page when a Perplexity user asks a question, so the answer can link to it. | Not always (user-requested) | Perplexity says user fetches generally ignore robots.txt, so a firewall is the only real block. |
How Perplexity finds and cites websites
Perplexity answers questions with numbered citations that link to the sources it used, so being crawlable is the entry ticket to being one of those links.
PerplexityBot is designed to surface and link websites in Perplexity's search results. Perplexity says it is not used to crawl content for AI foundation models, and recommends allowing it in robots.txt.
Perplexity-User visits a page when someone asks Perplexity a question, so the answer can include and link to it. Perplexity says it is not used for web crawling or for training, and that because a person requested the fetch, it generally ignores robots.txt rules.
Perplexity says each setting works independently and that it may take up to 24 hours for its systems to reflect a change.
How to allow or block Perplexity in robots.txt
Perplexity-User generally ignores robots.txt, so robots rules mainly control PerplexityBot. A crawler with its own group ignores your User-agent: * group, so repeat any other rules you still want it to follow (such as Disallow: /wp-admin/) inside its group.
Appear in Perplexity (recommended)
Perplexity's own docs recommend allowing PerplexityBot. A permissive User-agent: * group already covers it.
User-agent: PerplexityBot
Allow: /Remove yourself from Perplexity search
Blocks the index crawler. Perplexity-User may still fetch pages people ask about, so use a firewall rule for a hard block.
User-agent: PerplexityBot
Disallow: /How to tell if it's really Perplexity, and keep your firewall from blocking it
Perplexity publishes the IP ranges for both agents: perplexitybot.json and perplexity-user.json. Match them against your logs, or allow them in your firewall.
Its documentation includes firewall setup notes for Cloudflare and AWS, and one warning worth knowing: Cloudflare's Bot Fight Mode can't be skipped with WAF custom rules. If Bot Fight Mode is on, Perplexity can be blocked even when your robots.txt and allow rules say otherwise.
Why Perplexity might not see your site
Cloudflare Bot Fight Mode
Perplexity notes it can't be bypassed with WAF custom rules, so it can block PerplexityBot regardless of your other settings.
Expecting robots.txt to stop Perplexity-User
It generally ignores robots.txt because a person asked for the page. Use a firewall rule if you need a hard block.
Content that only appears with JavaScript
Vercel's 2024 analysis found PerplexityBot doesn't run JavaScript. Keep your key text in the server HTML.
Waiting for changes to show
Perplexity says changes can take up to 24 hours. Recheck the next day before changing anything else.
Readable by Perplexity is not the same as recommended
Passing this check means Perplexity can read your site. Whether it cites or recommends you depends on what it finds: clear answers to your customers' questions, consistent business details and mentions on sites it trusts. That's the work of generative engine optimisation (GEO) and AI SEO, built on solid technical SEO. Our AI search team does exactly this for Australian businesses.
Readable by Perplexity is step one. Let's get it recommending you.
Book a free 30-minute AI search strategy session. We'll check where your business shows up, or doesn't, in ChatGPT, Gemini, Perplexity and Google AI Overviews, show you what your competitors are doing, and hand you a written plan to fix it.
- AI visibility review across the major engines
- Competitor analysis: who AI recommends instead of you
- Technical fixes from your crawler check, prioritised
- A written plan that's yours to keep. No obligation.
Apply for your free strategy call
Get your competitor analysis, deep dive and clear report. Takes about 30 seconds, no obligation.
Perplexity crawler questions, answered.
Does Perplexity use my content to train AI models?
Perplexity says PerplexityBot is not used to crawl content for AI foundation models, and that Perplexity-User is not used for training either.
Can I stop Perplexity-User with robots.txt?
Not reliably. Perplexity says that because a user requested the fetch, Perplexity-User generally ignores robots.txt. A firewall rule based on its published IP ranges is the way to block it.
My robots.txt allows PerplexityBot. Why is it still blocked?
Usually a firewall or CDN setting. Perplexity specifically notes that Cloudflare's Bot Fight Mode can't be skipped with WAF custom rules. The firewall test on this page shows whether PerplexityBot gets a block or challenge.
How long until Perplexity sees my robots.txt change?
Perplexity says it may take up to 24 hours for its systems to reflect changes.
Sources
Facts on this page were checked against these sources in November 2026. AI crawlers change often, so we review this page regularly.
Check another site, or a competitor's.
See whether Perplexity and every other AI crawler can read any website. Free, with no sign-up.
Run the Perplexity check