Free AI Crawler Checker
Can ChatGPT, Google AI Overviews, Perplexity and Claude actually read your website? Enter your address to test 20 AI crawlers live: robots.txt rules, firewall blocks, indexing and content they can see. Results in about 10 seconds.
Check your home page or any page URL. Free, no sign-up, and we don't save results.
- 7 AI search engines
- Live firewall test
- Exact robots.txt line
- Fix-it advice
Five checks most AI crawler tools skip.
Most free checkers only read robots.txt. That misses the firewall blocks and JavaScript-only content that quietly hide sites from AI search.
robots.txt rules, bot by bot
We read your live robots.txt the way crawlers do (RFC 9309), work out which group applies to each of 20 AI user agents, and show the exact line that allows or blocks it.
Firewall and CDN blocks
We request your page as each AI crawler and compare the response with a normal browser. A 403 or challenge page means a firewall is overriding your robots.txt.
Content without JavaScript
Most AI crawlers don't run JavaScript. We count the words in your raw HTML to show what they can actually read.
noindex and nosnippet
We check robots meta tags and X-Robots-Tag headers that keep a page out of Google's index, AI Overviews and AI Mode.
llms.txt and structured data
We look for an llms.txt file and schema.org markup, the extras that help AI systems understand who you are.
AI search crawlers vs AI training crawlers
"AI bots" are not one thing. Each AI company runs separate crawlers for separate jobs, and blocking the wrong one is the most common mistake we see on Australian business websites.
- AI search and answer crawlers (OAI-SearchBot, PerplexityBot, Claude-SearchBot, Googlebot, Bingbot) index your pages so AI engines can quote and link to you. Block these and you disappear from AI answers.
- User-requested fetchers (ChatGPT-User, Claude-User, Perplexity-User) open a page live when someone asks an assistant about it. Some don't follow robots.txt because a person asked for the page.
- Training crawlers (GPTBot, ClaudeBot, CCBot, Meta-ExternalAgent) collect content that may train future models. Blocking them is a legitimate choice and does not remove you from AI search.
- Opt-out tokens (Google-Extended, Applebot-Extended) aren't crawlers at all. They tell Google and Apple how they may use content their existing crawlers already collect.
Two examples make this concrete. OpenAI's crawler documentation says OAI-SearchBot and GPTBot are controlled independently, so you can appear in ChatGPT search while opting out of training. And Google says Google-Extended does not affect inclusion in Google Search, while AI Overviews and AI Mode follow your normal Search controls. Blocking Google-Extended won't keep you out of AI Overviews, and allowing it won't get you in.
Every AI crawler we check, and what it does
Sourced from each company's own crawler documentation where it exists. Last reviewed October 2026.
AI search and answer crawlers
Index your pages so AI search engines can find, quote and link to you. Blocking these is what removes you from AI answers.
| User agent | Company | What it does | If you block it | Follows robots.txt |
|---|---|---|---|---|
| Googlebot | Crawls for Google Search, including AI Overviews and AI Mode, which are controlled by the same Search settings. | You drop out of Google Search, AI Overviews and AI Mode. | Yes | |
| Bingbot | Microsoft | Crawls for Bing, which powers Microsoft Copilot answers and is used as a search source by other AI tools. | You lose Bing and Copilot visibility, and AI tools that lean on Bing results. | Yes |
| OAI-SearchBot | OpenAI | Surfaces websites in ChatGPT search answers and links. | Your pages will not appear in ChatGPT search answers (only as plain navigational links). | Yes |
| PerplexityBot | Perplexity | Indexes sites so Perplexity can surface and link to them in its answers. Not used for model training. | Perplexity is far less likely to cite or link to you. | Yes |
| Claude-SearchBot | Anthropic | Crawls to improve the relevance and accuracy of Claude's search results. | Your content is not indexed for Claude's search answers. | Yes |
| Applebot | Apple | Powers search in Siri, Spotlight and Safari, and can supply context for Apple's AI answers. | You lose visibility in Siri and Spotlight suggestions. | Yes |
| DuckAssistBot | DuckDuckGo | Fetches pages for DuckDuckGo's AI-assisted answers (DuckAssist). | DuckDuckGo's AI answers cannot draw on your pages. | Yes |
User-requested fetchers
Visit a page live when someone asks an AI assistant about it, or pastes your link into a chat.
| User agent | Company | What it does | If you block it | Follows robots.txt |
|---|---|---|---|---|
| ChatGPT-User | OpenAI | Visits a page when a ChatGPT user's question or action needs it. Does not crawl automatically. | ChatGPT may fail to open your page when a user asks about it. | Not always (user-requested) |
| Claude-User | Anthropic | Retrieves a page when a Claude user's question requires it. | Claude cannot read your page when a user asks about it. | Yes |
| Perplexity-User | Perplexity | Fetches a page when a Perplexity user asks a question, so the answer can link to it. | Perplexity says user fetches generally ignore robots.txt, so a firewall is the only real block. | Not always (user-requested) |
| Meta-ExternalFetcher | Meta | Fetches individual links when a Meta AI user requests them. | Meta AI may be unable to read your page on request. | Not always (user-requested) |
| MistralAI-User | Mistral | Fetches pages when a Le Chat user's question needs live web content. | Mistral's Le Chat cannot cite your page. | Yes |
AI training crawlers
Collect content that may be used to train AI models. Blocking these is a business choice and does not remove you from AI search.
| User agent | Company | What it does | If you block it | Follows robots.txt |
|---|---|---|---|---|
| GPTBot | OpenAI | Crawls content that may be used to train OpenAI's models. | Your content is excluded from future OpenAI training. ChatGPT search is unaffected (that is OAI-SearchBot). | Yes |
| ClaudeBot | Anthropic | Collects public web content that may be used to train Anthropic's models. | Your future content is excluded from Anthropic's training data. | Yes |
| Meta-ExternalAgent | Meta | Crawls the web to train Meta's AI models and index content for its products. | Your content is excluded from Meta's AI training crawl. | Yes |
| CCBot | Common Crawl | Builds the open Common Crawl dataset, which many AI models are trained on. | Your future pages are left out of the Common Crawl archive. | Yes |
| Amazonbot | Amazon | Crawls to improve Amazon's products and services, including Alexa answers, and may be used to train Amazon's AI models. | Amazon's crawler stops reading your site. | Yes |
| Bytespider | ByteDance | ByteDance's (TikTok's parent company) crawler, associated with its AI products. | ByteDance's crawler is asked not to read your site. | Yes |
Opt-out tokens (no crawler)
robots.txt controls read by an existing crawler. They change how your content is used, not whether it is crawled.
| User agent | Company | What it does | If you block it | Follows robots.txt |
|---|---|---|---|---|
| Google-Extended | Controls whether Google may use your content to train Gemini and to ground answers in Gemini Apps and Vertex AI. | No effect on Google Search, AI Overviews or rankings. Only Gemini training and grounding outside Search. | Read as a robots.txt token | |
| Applebot-Extended | Apple | Controls whether Apple may use content Applebot collects to train its foundation models. It does not crawl. | Siri and Spotlight search are unaffected. Only Apple's model training. | Read as a robots.txt token |
How to allow AI crawlers in robots.txt
Your robots.txt file lives at the root of your domain (for example
yourbusiness.com.au/robots.txt). Crawlers follow the most specific group that
names them, and fall back to the User-agent: * group if none does. Within a group,
the longest matching rule wins.
Option 1: welcome every AI crawler
If you want maximum visibility in AI search and don't mind your content being used for training, this is all you need:
User-agent: *
Allow: /
Sitemap: https://www.example.com.au/sitemap.xmlOption 2: allow AI search, block AI training
The middle path most businesses choose: stay visible in ChatGPT, Perplexity, Claude and Google AI answers, but opt out of model training.
# Everyone, including AI search crawlers, may crawl the site
User-agent: *
Allow: /
# Opt out of AI model training only
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Meta-ExternalAgent
User-agent: Bytespider
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /
Sitemap: https://www.example.com.au/sitemap.xmlCommon robots.txt mistakes that block AI search
- A leftover
Disallow: /from a staging site that went live. It blocks every crawler, including Googlebot. - Naming a bot without repeating your rules. Once a crawler has its own group, it ignores the
*group completely. Copy any rules you still want it to follow. - Blocking a whole company by mistake. Disallowing every OpenAI bot also removes you from ChatGPT search. Block GPTBot only if training is your concern.
- A robots.txt that errors. Under the robots.txt standard, a server error on
/robots.txtmeans crawlers must treat the whole site as blocked. - Plugin or platform defaults. Some security plugins and site builders add AI bot blocks automatically. Check your live file, not the one you think you published.
Changes aren't instant. OpenAI and Meta both say crawlers can cache robots.txt for up to about 24 hours.
Why robots.txt isn't the whole story
Firewalls and CDNs can override it
robots.txt is a polite request. A firewall decides who actually gets in. In July 2025 Cloudflare began asking new domains whether to allow AI crawlers, with blocking as the default, and many hosts and security plugins now offer one-click "block AI bots" switches. A site can say "welcome" in robots.txt while its firewall returns a 403. That's why this checker runs a live request as each AI crawler and compares it with a normal browser.
Content hidden behind JavaScript
Vercel's 2024 study of AI crawler traffic found that the major AI crawlers fetch JavaScript files but don't run them. If your services, prices or contact details only appear after scripts load, as on many single-page apps and some page builders, AI engines may see an almost empty page. Server-side rendering or static HTML fixes it. It is one reason we build client sites on fast, static-first frameworks.
noindex and nosnippet
Google's AI features only draw on pages that are indexed and eligible for snippets. A stray
noindex or nosnippet in a meta tag or HTTP header removes a page from
AI Overviews and AI Mode even when Googlebot can crawl it.
Readable is not the same as recommended
Passing this check means AI engines can read your site. Whether they cite or recommend you depends on what they find: clear, specific answers, consistent business details, structured data and mentions on sites they trust. That's the work of generative engine optimisation (GEO) and AI SEO, and it builds on solid technical SEO. For a practical start, read how brands get found in AI search, how to structure content for AI search and schema markup for local businesses. Then track whether AI search is sending you customers.
Crawlable is step one. Let's get AI recommending you.
Book a free 30-minute AI search strategy session. We'll check where your business shows up, or doesn't, in ChatGPT, Gemini, Perplexity and Google AI Overviews, show you what your competitors are doing, and hand you a written plan to fix it.
- AI visibility review across the major engines
- Competitor analysis: who AI recommends instead of you
- Technical fixes from your crawler check, prioritised
- A written plan that's yours to keep. No obligation.
Apply for your free strategy call
Get your competitor analysis, deep dive and clear report. Takes about 30 seconds, no obligation.
AI crawler questions, answered.
What is an AI crawler?
An AI crawler is an automated bot that reads web pages for an AI company. Some index pages so AI search engines like ChatGPT search, Perplexity and Google's AI Overviews can quote and link to them. Some fetch a page live when a user asks about it. Others collect content to train AI models. Each one identifies itself with a user agent token, such as OAI-SearchBot or GPTBot, that you can allow or block in robots.txt.
Does blocking GPTBot remove my site from ChatGPT?
No. GPTBot is OpenAI's training crawler. Whether you appear in ChatGPT search answers is controlled by a separate bot, OAI-SearchBot, and OpenAI says the two settings are independent. You can block GPTBot to opt out of training and still allow OAI-SearchBot so ChatGPT can cite you.
How do I make sure Google AI Overviews and AI Mode can use my site?
Google says AI Overviews and AI Mode are part of Search, so they follow your normal Googlebot rules. Googlebot must be allowed in robots.txt, the page must be indexable (no noindex), and nosnippet must not be set. Blocking Google-Extended does not affect Search or AI Overviews. It only covers Gemini training and grounding outside Search.
Should I block AI training crawlers?
It is a business decision, not an SEO requirement. Blocking training crawlers such as GPTBot, ClaudeBot and CCBot keeps your future content out of model training, but it does not remove you from AI search. Most businesses that want leads from AI allow every search and answer crawler. Publishers with paywalled or licensed content often block training. This checker scores AI search access only, so blocking training bots won't lower your score.
My robots.txt allows AI crawlers, so why does the checker show a firewall block?
robots.txt is a request, not an access control. Firewalls, CDNs and hosting security tools can block AI bots before they ever read it. Cloudflare, for example, began asking new domains in July 2025 whether to allow AI crawlers, with blocking as the default. Some firewalls also block any request that claims to be a bot but doesn't come from that bot's official IP range, so check the setting in your firewall dashboard to confirm.
Can AI crawlers read JavaScript?
Mostly not. A 2024 analysis of AI crawler traffic by Vercel found that the major AI crawlers, including OpenAI's and Anthropic's, fetch JavaScript files but don't execute them. Google's crawler is the main exception. If your key content only appears after JavaScript runs, many AI engines can't see it. Server-side rendering or static HTML fixes this.
What is llms.txt and do I need one?
llms.txt is a proposed standard: a plain-text Markdown file at /llms.txt that summarises your site for AI tools. It is cheap to add, but the major AI search engines haven't confirmed they use it, so it is a nice-to-have. Crawler access, indexable pages and clear content matter far more.
Is the AI crawler checker free?
Yes, completely free, with no sign-up and no email required. We don't save the reports. Each check reads only public files and pages (robots.txt, llms.txt and the page you enter), the same way a crawler would.
How often should I check my AI crawler access?
Check after any website rebuild, hosting or CDN change, security plugin update or robots.txt edit, and at least once a quarter. New AI crawlers appear regularly, and a single setting change can quietly block them.
Check another site, or a competitor's.
See which AI crawlers can read any website in seconds. Free, with no sign-up.
Run the AI crawler checker