ChatGPT Crawler Checker
Can ChatGPT find, open and cite your website? Test OpenAI's three crawlers live: OAI-SearchBot (ChatGPT search), ChatGPT-User (pages ChatGPT opens for people) and GPTBot (training), plus your firewall and page signals.
Check your home page or any page URL. Free, no sign-up, and we don't save results.
- OpenAI's crawlers first
- Live robots.txt check
- Firewall test
- Fix-it advice
OpenAI and ChatGPT: every crawler at a glance
| User agent | What it does | Follows robots.txt | If you block it |
|---|---|---|---|
| OAI-SearchBot | Surfaces websites in ChatGPT search answers and links. | Yes | Your pages will not appear in ChatGPT search answers (only as plain navigational links). |
| ChatGPT-User | Visits a page when a ChatGPT user's question or action needs it. Does not crawl automatically. | Not always (user-requested) | ChatGPT may fail to open your page when a user asks about it. |
| GPTBot | Crawls content that may be used to train OpenAI's models. | Yes | Your content is excluded from future OpenAI training. ChatGPT search is unaffected (that is OAI-SearchBot). |
| OAI-AdsBot Not tested by this checker | Checks the safety of landing pages submitted as ads on ChatGPT. OpenAI says it only visits pages submitted as ads. | Not stated | Only relevant if you advertise on ChatGPT. |
How ChatGPT finds and cites websites
OpenAI runs separate crawlers for separate jobs, and each one is controlled on its own in robots.txt. Understanding that split is the most important step before you change anything.
OAI-SearchBot decides whether you can appear in ChatGPT search. OpenAI says sites that opt out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as plain navigational links. If you want ChatGPT to cite and link to your business, this is the crawler to allow.
ChatGPT-User is not an automatic crawler. It visits a page when a ChatGPT user's request needs it, for example when someone pastes your link or a Custom GPT calls your site. OpenAI says it is not used to decide whether content appears in search, and because a person started the action, robots.txt rules may not apply.
GPTBot collects content that may be used to train OpenAI's models. Blocking it opts your future content out of training and has no effect on ChatGPT search. OpenAI says the settings are independent, and that if you allow both GPTBot and OAI-SearchBot it may use a single crawl for both to avoid crawling twice.
After a robots.txt change, OpenAI says it can take about 24 hours for ChatGPT search to adjust.
How to allow or block ChatGPT in robots.txt
Pick the setup that matches what you want from ChatGPT. A crawler with its own group ignores your User-agent: * group, so repeat any other rules you still want it to follow (such as Disallow: /wp-admin/) inside its group.
Appear in ChatGPT search, opt out of training
The most common choice for businesses that want leads from ChatGPT.
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: GPTBot
Disallow: /Welcome all of OpenAI's crawlers
Maximum visibility. A permissive User-agent: * group already does this unless something else blocks them.
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: GPTBot
Allow: /Block OpenAI completely
This also removes you from ChatGPT search answers. ChatGPT-User may still fetch pages people ask for, so use a firewall rule if you need a hard block.
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: GPTBot
Disallow: /How to tell if it's really OpenAI
User agents are easy to fake. OpenAI publishes the IP ranges each crawler uses, so you can match requests in your server or CDN logs against them: searchbot.json (OAI-SearchBot), chatgpt-user.json (ChatGPT-User), gptbot.json (GPTBot) and adsbot.json (OAI-AdsBot).
If you allow OpenAI through a firewall, allow these published ranges (or your CDN's verified-bot category) rather than any request that simply claims to be GPTBot.
Why ChatGPT might not see your site
Blocking every OpenAI bot to stop training
Only GPTBot is used for training. A blanket OpenAI block also removes you from ChatGPT search, which is usually not what a business wants.
A firewall that blocks AI bots
CDN and security settings can return a 403 to OpenAI's crawlers even when robots.txt allows them. The firewall test above shows whether that's happening.
Content that only appears with JavaScript
Vercel's 2024 analysis found OpenAI's crawlers fetch JavaScript files but don't run them. Keep your key text in the HTML the server sends.
A new named group that drops your other rules
Adding a GPTBot or OAI-SearchBot group means that crawler ignores your * group. Repeat any Disallow rules you still need.
Readable by ChatGPT is not the same as recommended
Passing this check means ChatGPT can read your site. Whether it cites or recommends you depends on what it finds: clear answers to your customers' questions, consistent business details and mentions on sites it trusts. That's the work of generative engine optimisation (GEO) and AI SEO, built on solid technical SEO. Our AI search team does exactly this for Australian businesses.
Readable by ChatGPT is step one. Let's get it recommending you.
Book a free 30-minute AI search strategy session. We'll check where your business shows up, or doesn't, in ChatGPT, Gemini, Perplexity and Google AI Overviews, show you what your competitors are doing, and hand you a written plan to fix it.
- AI visibility review across the major engines
- Competitor analysis: who AI recommends instead of you
- Technical fixes from your crawler check, prioritised
- A written plan that's yours to keep. No obligation.
Apply for your free strategy call
Get your competitor analysis, deep dive and clear report. Takes about 30 seconds, no obligation.
ChatGPT crawler questions, answered.
Does blocking GPTBot remove my site from ChatGPT?
No. GPTBot is OpenAI's training crawler. Appearing in ChatGPT search answers is controlled by OAI-SearchBot, and OpenAI says each setting is independent. You can block GPTBot and still be cited in ChatGPT search.
How long does ChatGPT take to pick up a robots.txt change?
OpenAI says it can take about 24 hours from a robots.txt update for its search systems to adjust.
Can ChatGPT read my page if robots.txt blocks ChatGPT-User?
Possibly. OpenAI says that because ChatGPT-User actions are started by a person, robots.txt rules may not apply. If you need to stop it completely, use a firewall rule matched to OpenAI's published ChatGPT-User IP ranges.
If OAI-SearchBot can crawl my site, will ChatGPT recommend my business?
Not automatically. Crawler access makes you eligible. Whether ChatGPT cites you depends on how clearly your pages answer the question, how consistent your business details are across the web and how much trusted sites mention you. That is the work of generative engine optimisation.
What is OAI-AdsBot?
It checks landing pages submitted as ads on ChatGPT for safety, and OpenAI says it only visits pages submitted as ads. It is irrelevant unless you advertise on ChatGPT.
Sources
Facts on this page were checked against these sources in November 2026. AI crawlers change often, so we review this page regularly.
Check another site, or a competitor's.
See whether ChatGPT and every other AI crawler can read any website. Free, with no sign-up.
Run the ChatGPT check