Gemini & Google AI Overviews Crawler Checker
Can Google use your website in AI Overviews, AI Mode and Gemini? Test your Googlebot and Google-Extended rules, plus the noindex and snippet settings that decide what Google's AI features can show.
Check your home page or any page URL. Free, no sign-up, and we don't save results.
- Google's crawlers first
- Live robots.txt check
- Every AI crawler too
- Fix-it advice
Google and Google AI: every crawler at a glance
| User agent | What it does | Follows robots.txt | If you block it |
|---|---|---|---|
| Googlebot | Crawls for Google Search, including AI Overviews and AI Mode, which are controlled by the same Search settings. | Yes | You drop out of Google Search, AI Overviews and AI Mode. |
| Google-Extended | Controls whether Google may use your content to train Gemini and to ground answers in Gemini Apps and Vertex AI. | Read as a robots.txt token | No effect on Google Search, AI Overviews or rankings. Only Gemini training and grounding outside Search. |
How Google's AI features use your website
Google has two very different controls here, and mixing them up is common.
AI Overviews and AI Mode are part of Google Search. Google says you manage them through Googlebot and normal Search controls, not a separate AI setting. Googlebot must be allowed in robots.txt and the page must be indexable. To limit what's shown, Google points to nosnippet, data-nosnippet, max-snippet and noindex, which also affect your ordinary search results.
Google-Extended controls Gemini, not Search. It is a robots.txt token, not a crawler. Google says it manages whether content it crawls may be used to train future Gemini models (in Gemini Apps and the Vertex AI API) and for grounding in Gemini Apps and Grounding with Google Search on Vertex AI. Google is explicit that it does not affect inclusion in Google Search and is not a ranking signal.
Google-Extended has no user agent of its own. Crawling is done by Google's existing crawlers, and the token only governs how the content may be used. Google also says it intends to support the AIPREF training preference signal (user-agent: __usage_ai-training) in early 2027.
Unlike most AI crawlers, Googlebot renders JavaScript, so Google's AI features can see content that other AI crawlers miss.
How to allow or block Google AI in robots.txt
Never block Googlebot unless you want to leave Google Search entirely. Use Google-Extended for Gemini, and snippet controls for AI Overviews.
Stay in Search and AI Overviews, opt out of Gemini training and grounding
Blocking Google-Extended also stops grounding in Gemini Apps, so Gemini's own answers can't draw on your pages that way.
User-agent: Googlebot
Allow: /
User-agent: Google-Extended
Disallow: /Keep one section out of AI Overviews and snippets
data-nosnippet hides marked content from snippets while the rest of the page stays eligible.
<div data-nosnippet>
Pricing details we don't want quoted.
</div>Keep a whole page out of AI Overviews
nosnippet removes the page's text from all snippets, including normal results, so use it sparingly.
<meta name="robots" content="nosnippet">How to tell if it's really Googlebot
Google's documented method: run a reverse DNS lookup on the requesting IP, check the hostname ends in googlebot.com, google.com or googleusercontent.com, then run a forward lookup on that hostname and confirm it returns the original IP.
Google also publishes its crawler IP ranges, including common-crawlers.json for Googlebot. That's why this checker doesn't spoof Googlebot in its firewall test: well-configured firewalls rightly block fake Googlebot requests.
Why Google AI might not see your site
Blocking Google-Extended to leave AI Overviews
It doesn't. AI Overviews follow Googlebot and Search controls. Google-Extended only covers Gemini training and grounding outside Search.
A plugin adding noindex or nosnippet
SEO and staging plugins sometimes add these sitewide. They remove pages from AI Overviews and AI Mode as well as normal results.
Blocking CSS and JavaScript files
Googlebot renders pages. If robots.txt blocks your scripts or stylesheets, Google may not see the page the way visitors do.
Treating snippet controls as free
nosnippet and max-snippet also shrink your normal search snippets, which can lower click-through. Use data-nosnippet for specific sections instead.
Readable by Google AI is not the same as recommended
Passing this check means Google AI can read your site. Whether it cites or recommends you depends on what it finds: clear answers to your customers' questions, consistent business details and mentions on sites it trusts. That's the work of generative engine optimisation (GEO) and AI SEO, built on solid technical SEO. Our AI search team does exactly this for Australian businesses.
Readable by Google AI is step one. Let's get it recommending you.
Book a free 30-minute AI search strategy session. We'll check where your business shows up, or doesn't, in ChatGPT, Gemini, Perplexity and Google AI Overviews, show you what your competitors are doing, and hand you a written plan to fix it.
- AI visibility review across the major engines
- Competitor analysis: who AI recommends instead of you
- Technical fixes from your crawler check, prioritised
- A written plan that's yours to keep. No obligation.
Apply for your free strategy call
Get your competitor analysis, deep dive and clear report. Takes about 30 seconds, no obligation.
Google AI crawler questions, answered.
Does blocking Google-Extended remove my site from AI Overviews?
No. Google says AI Overviews and AI Mode are part of Search and follow Googlebot and Search controls. Google-Extended only affects Gemini model training and grounding in Gemini Apps and Vertex AI, and doesn't affect Search inclusion or ranking.
How do I keep my content out of AI Overviews?
Google points to its snippet and indexing controls: nosnippet, data-nosnippet, max-snippet and noindex. They also affect normal search results, so data-nosnippet on specific sections is usually the least costly option.
Is there a Gemini crawler I can block?
There is no separate Gemini user agent. Google-Extended is a robots.txt token read by Google's existing crawlers, and it controls whether your content is used for Gemini training and grounding.
Can Google's AI features read JavaScript content?
Yes. Googlebot renders JavaScript, unlike most other AI crawlers. Google still recommends not blocking the CSS and JavaScript files it needs to render your pages.
Sources
- AI features and your website (Google Search Central)
- Google's common crawlers, including Google-Extended (Google Search Central)
- Verifying Googlebot and other Google crawlers (Google Search Central)
Facts on this page were checked against these sources in November 2026. AI crawlers change often, so we review this page regularly.
Check another site, or a competitor's.
See whether Google AI and every other AI crawler can read any website. Free, with no sign-up.
Run the Google AI check