Claude Crawler Checker
Can Claude find and read your website? Test Anthropic's three crawlers live: Claude-SearchBot (Claude's search results), Claude-User (pages Claude opens for people) and ClaudeBot (training), plus your firewall and page signals.
Check your home page or any page URL. Free, no sign-up, and we don't save results.
- Anthropic's crawlers first
- Live robots.txt check
- Firewall test
- Fix-it advice
Anthropic and Claude: every crawler at a glance
| User agent | What it does | Follows robots.txt | If you block it |
|---|---|---|---|
| Claude-SearchBot | Crawls to improve the relevance and accuracy of Claude's search results. | Yes | Your content is not indexed for Claude's search answers. |
| Claude-User | Retrieves a page when a Claude user's question requires it. | Yes | Claude cannot read your page when a user asks about it. |
| ClaudeBot | Collects public web content that may be used to train Anthropic's models. | Yes | Your future content is excluded from Anthropic's training data. |
How Claude finds and reads websites
Anthropic uses three user agents, and its help article says all of them respect "do not crawl" signals by honouring standard robots.txt directives. That makes Claude one of the more predictable platforms to control.
Claude-SearchBot crawls to improve the relevance and accuracy of Claude's search results. Blocking it stops your content being indexed for that purpose.
Claude-User retrieves a page when a Claude user's question needs it. Anthropic says blocking it may reduce your visibility in user-directed web search.
ClaudeBot collects public web content that may be used to train Anthropic's models. Blocking it tells Anthropic not to use your future content for training, and it doesn't stop Claude-SearchBot or Claude-User.
Anthropic also supports the non-standard Crawl-delay rule, so you can slow ClaudeBot down instead of blocking it. Its help article (updated October 2026) adds that it now honours a machine-readable "no AI training" preference based on an IETF draft, and that a robots.txt rule naming ClaudeBot takes priority over it.
Older guides mention tokens such as anthropic-ai and Claude-Web. Anthropic's current help article lists only ClaudeBot, Claude-User and Claude-SearchBot, so write your rules for those.
How to allow or block Claude in robots.txt
Choose the setup that matches what you want from Claude. A crawler with its own group ignores your User-agent: * group, so repeat any other rules you still want it to follow (such as Disallow: /wp-admin/) inside its group.
Appear in Claude's search, opt out of training
Keeps you visible when people use Claude to research, without contributing to training.
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: ClaudeBot
Disallow: /Slow ClaudeBot down instead of blocking it
Anthropic supports Crawl-delay (in seconds between requests). Useful if crawl volume is your concern, not training.
User-agent: ClaudeBot
Crawl-delay: 1Block Anthropic completely
Removes you from Claude's search results and stops Claude reading your pages on request.
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: ClaudeBot
Disallow: /How to tell if it's really Anthropic
Anthropic links to a list of the source IP addresses its crawlers use, at claude.com/crawling/bots.json. A request from one of those addresses is genuinely from Anthropic.
Use robots.txt rather than IP blocking to opt out. Anthropic warns that blocking its IP addresses may not work correctly or persistently guarantee an opt-out, because it stops the crawler reading your robots.txt.
Why Claude might not see your site
Blocking ClaudeBot and expecting Claude to forget you
ClaudeBot is the training crawler. Claude-SearchBot and Claude-User are separate, so Claude can still find and read your site.
Only using the old tokens
Rules for anthropic-ai or Claude-Web alone don't cover the three user agents Anthropic documents today.
Blocking by IP address
Anthropic says IP blocks can stop it reading robots.txt, so your opt-out may not stick. Use robots.txt.
Content that only appears with JavaScript
Vercel's 2024 analysis found Anthropic's crawler fetches JavaScript files but doesn't run them. Keep key text in the server HTML.
Readable by Claude is not the same as recommended
Passing this check means Claude can read your site. Whether it cites or recommends you depends on what it finds: clear answers to your customers' questions, consistent business details and mentions on sites it trusts. That's the work of generative engine optimisation (GEO) and AI SEO, built on solid technical SEO. Our AI search team does exactly this for Australian businesses.
Readable by Claude is step one. Let's get it recommending you.
Book a free 30-minute AI search strategy session. We'll check where your business shows up, or doesn't, in ChatGPT, Gemini, Perplexity and Google AI Overviews, show you what your competitors are doing, and hand you a written plan to fix it.
- AI visibility review across the major engines
- Competitor analysis: who AI recommends instead of you
- Technical fixes from your crawler check, prioritised
- A written plan that's yours to keep. No obligation.
Apply for your free strategy call
Get your competitor analysis, deep dive and clear report. Takes about 30 seconds, no obligation.
Claude crawler questions, answered.
Does blocking ClaudeBot remove my site from Claude's answers?
No. ClaudeBot is Anthropic's training crawler. Claude-SearchBot (search) and Claude-User (pages Claude opens for people) are controlled separately, so blocking ClaudeBot alone doesn't stop Claude finding or reading your site.
Does Anthropic respect robots.txt?
Yes. Anthropic's help article says its bots honour standard robots.txt directives, and that applies to ClaudeBot, Claude-User and Claude-SearchBot.
Can I slow ClaudeBot down instead of blocking it?
Yes. Anthropic supports the non-standard Crawl-delay rule. For example, Crawl-delay: 1 in the ClaudeBot group asks it to wait a second between requests.
Should I still block anthropic-ai and Claude-Web?
They don't hurt, but they aren't enough. Anthropic's current documentation lists ClaudeBot, Claude-User and Claude-SearchBot, so make sure your rules name those.
Why shouldn't I block Anthropic by IP address?
Anthropic says IP blocking may not work correctly or persistently guarantee an opt-out, because it impedes the crawler's ability to read your robots.txt. A robots.txt rule is the reliable way to opt out.
Sources
- Does Anthropic crawl data from the web, and how can site owners block the crawler? (Anthropic)
- The rise of the AI crawler (Vercel, December 2024)
Facts on this page were checked against these sources in November 2026. AI crawlers change often, so we review this page regularly.
Check another site, or a competitor's.
See whether Claude and every other AI crawler can read any website. Free, with no sign-up.
Run the Claude check