Grok Crawler Checker
Can Grok reach your website? xAI hasn't published any crawler documentation, so this checker tests your robots.txt against the names most often reported for Grok, alongside every documented AI crawler, and tells you honestly what that can and can't prove.
Check your home page or any page URL. Free, no sign-up, and we don't save results.
- xAI's crawlers first
- Live robots.txt check
- Every AI crawler too
- Fix-it advice
xAI and Grok: every crawler at a glance
| User agent | What it does | Follows robots.txt | If you block it |
|---|---|---|---|
| GrokBot | Reported by third-party directories as a crawler for xAI's Grok. Not documented by xAI. | Unknown (undocumented) | Unknown. xAI has not said whether it honours rules for this name. |
| xAI-Grok | Another name third-party directories report for xAI's crawler. Not documented by xAI. | Unknown (undocumented) | Unknown. xAI has not said whether it honours rules for this name. |
| xAI-SearchBot | A search crawler name some directories attribute to xAI. Not documented by xAI. | Unknown (undocumented) | Unknown. xAI has not said whether it honours rules for this name. |
What we know, and don't know, about Grok's crawlers
Grok is xAI's AI assistant, available in X and in xAI's own apps. Unlike OpenAI, Anthropic, Google, Apple, Meta and Perplexity, xAI has not published crawler documentation: as of November 2026 we could find no official list of user agents, no robots.txt guidance and no IP ranges.
Third-party bot directories report several names for xAI's crawlers, including GrokBot, xAI-Grok and xAI-SearchBot, and some describe a separate browsing agent. They disagree with each other, and none of these names is confirmed by xAI.
That's why this page is different from most Grok checkers. We test your robots.txt against the three most commonly reported names and show whether they're allowed, but we don't pretend those rules are guaranteed to work, and we don't run a firewall test because there's no official user agent to simulate.
The rest of the report covers every documented AI crawler, because the fundamentals that help Grok also help every other AI engine.
How to allow or block Grok in robots.txt
Rules for undocumented names are a best effort. If you want visibility, you don't need to add anything: a permissive User-agent: * group covers any crawler that honours robots.txt.
Ask Grok's reported crawlers to stay out (best effort)
xAI hasn't confirmed these names or whether it honours them. Use a firewall rule on user agents you actually see in your logs for a firm block.
User-agent: GrokBot
User-agent: xAI-Grok
User-agent: xAI-SearchBot
Disallow: /Stay open to Grok and every other AI
No special rules needed. Just make sure nothing else in robots.txt or your firewall blocks AI crawlers.
User-agent: *
Allow: /
Sitemap: https://www.example.com.au/sitemap.xmlHow to see whether Grok is visiting
Check your server or CDN logs for user agents containing "grok" or "xai". That's the only reliable way to see what actually visits your site while xAI's crawlers are undocumented.
If you need a firm block, create a firewall rule for the user agents you observe and recheck every few months, because undocumented names can change without notice.
Why Grok might not see your site
Treating unofficial tokens as guaranteed
Directories list different names and xAI hasn't confirmed any. A robots.txt rule may not be honoured.
Copying long "block all AI" lists
Generic lists often include tokens for crawlers you want, such as OAI-SearchBot or PerplexityBot. Block deliberately, crawler by crawler.
Over-broad firewall rules
Blocking every unrecognised bot can also catch documented AI search crawlers. Check the full report below after any change.
Content that only appears with JavaScript
Most AI crawlers don't run JavaScript. Keep key text in the server HTML so any crawler that reaches you can read it.
Readable by Grok is not the same as recommended
Passing this check means Grok can read your site. Whether it cites or recommends you depends on what it finds: clear answers to your customers' questions, consistent business details and mentions on sites it trusts. That's the work of generative engine optimisation (GEO) and AI SEO, built on solid technical SEO. Our AI search team does exactly this for Australian businesses.
Readable by Grok is step one. Let's get it recommending you.
Book a free 30-minute AI search strategy session. We'll check where your business shows up, or doesn't, in ChatGPT, Gemini, Perplexity and Google AI Overviews, show you what your competitors are doing, and hand you a written plan to fix it.
- AI visibility review across the major engines
- Competitor analysis: who AI recommends instead of you
- Technical fixes from your crawler check, prioritised
- A written plan that's yours to keep. No obligation.
Apply for your free strategy call
Get your competitor analysis, deep dive and clear report. Takes about 30 seconds, no obligation.
Grok crawler questions, answered.
Does Grok have an official web crawler?
As of November 2026, xAI hasn't published documentation naming its crawlers, the rules they follow or their IP ranges. Third-party directories report names such as GrokBot, xAI-Grok and xAI-SearchBot, but xAI hasn't confirmed them.
Can I block Grok with robots.txt?
You can add rules for the reported names, but it's a best effort because xAI hasn't said whether it honours them. For a firm block, use a firewall rule based on the user agents you actually see in your server logs.
How do I get my business mentioned by Grok?
The same fundamentals that work for every AI engine: let crawlers reach your site, keep key content in the HTML, answer customer questions clearly and build a consistent presence across the web, including on X where Grok is built in.
Why don't you run a firewall test for Grok?
Because there's no official Grok user agent to simulate. Testing a made-up user agent would produce results that look precise but prove nothing.
Check another site, or a competitor's.
See whether Grok and every other AI crawler can read any website. Free, with no sign-up.
Run the Grok check