Meta AI Crawler Checker
Can Meta's AI crawlers reach your website? Test Meta-ExternalAgent (AI training and indexing) and Meta-ExternalFetcher (links Meta AI opens for people) live, without breaking the link previews your social posts rely on.
Check your home page or any page URL. Free, no sign-up, and we don't save results.
- Meta's crawlers first
- Live robots.txt check
- Firewall test
- Fix-it advice
Meta and Meta AI: every crawler at a glance
| User agent | What it does | Follows robots.txt | If you block it |
|---|---|---|---|
| Meta-ExternalAgent | Crawls the web to train Meta's AI models and index content for its products. | Yes | Your content is excluded from Meta's AI training crawl. |
| Meta-ExternalFetcher | Fetches individual links when a Meta AI user requests them. | Not always (user-requested) | Meta AI may be unable to read your page on request. |
| facebookexternalhit Not tested by this checker | Fetches pages shared on Facebook, Instagram and Messenger to build the link preview (title, description and image). | May bypass robots.txt for security or integrity checks | Your links show without a proper preview when shared. |
How Meta's crawlers use your website
Meta documents three crawlers, and only two of them are about AI.
Meta-ExternalAgent crawls the web to train AI models or to improve products by indexing content directly. Meta says it is controlled through robots.txt, so a Disallow rule for its token opts you out.
Meta-ExternalFetcher fetches individual links when a user requests them, supporting agentic AI features. Because the fetch is user-requested, Meta says it may bypass robots.txt.
facebookexternalhit is the long-standing crawler that builds link previews when your pages are shared on Facebook, Instagram or Messenger. It isn't an AI crawler, and blocking it breaks your previews.
Meta notes that crawlers may cache your robots.txt for up to 24 hours, so changes can take a day to apply.
How to allow or block Meta AI in robots.txt
Write rules for Meta's AI crawlers without touching facebookexternalhit. A crawler with its own group ignores your User-agent: * group, so repeat any other rules you still want it to follow (such as Disallow: /wp-admin/) inside its group.
Opt out of Meta AI training, keep link previews
facebookexternalhit stays allowed through your * group, so shared links still show previews.
User-agent: meta-externalagent
Disallow: /Welcome Meta's AI crawlers
A permissive * group already allows them. Use this only if another rule might catch them.
User-agent: meta-externalagent
User-agent: meta-externalfetcher
Allow: /Why Meta AI might not see your site
Blocking facebookexternalhit by mistake
It powers link previews, not AI. Blocking it makes your shared links look broken on Facebook, Instagram and Messenger.
Expecting robots.txt to stop Meta-ExternalFetcher
Meta says it may bypass robots.txt because a person requested the link. Use a firewall rule for a hard block.
A firewall that blocks AI bots
Bot protection can return 403s to Meta's crawlers even when robots.txt allows them. The firewall test above shows this.
Content that only appears with JavaScript
Vercel's 2024 analysis found Meta-ExternalAgent doesn't run JavaScript. Keep key text in the server HTML.
Readable by Meta AI is not the same as recommended
Passing this check means Meta AI can read your site. Whether it cites or recommends you depends on what it finds: clear answers to your customers' questions, consistent business details and mentions on sites it trusts. That's the work of generative engine optimisation (GEO) and AI SEO, built on solid technical SEO. Our AI search team does exactly this for Australian businesses.
Readable by Meta AI is step one. Let's get it recommending you.
Book a free 30-minute AI search strategy session. We'll check where your business shows up, or doesn't, in ChatGPT, Gemini, Perplexity and Google AI Overviews, show you what your competitors are doing, and hand you a written plan to fix it.
- AI visibility review across the major engines
- Competitor analysis: who AI recommends instead of you
- Technical fixes from your crawler check, prioritised
- A written plan that's yours to keep. No obligation.
Apply for your free strategy call
Get your competitor analysis, deep dive and clear report. Takes about 30 seconds, no obligation.
Meta AI crawler questions, answered.
Will blocking Meta-ExternalAgent break my Facebook link previews?
No. Link previews come from a different crawler, facebookexternalhit. Block meta-externalagent to opt out of AI training and leave facebookexternalhit allowed.
Does Meta use my website to train AI?
Meta says Meta-ExternalAgent crawls the web to train AI models or improve products by indexing content directly, and that you can control it with robots.txt.
Can I stop Meta AI opening my pages when someone asks?
Not reliably with robots.txt. Meta says Meta-ExternalFetcher may bypass robots.txt because the fetch is user-requested, so a firewall rule is the way to block it.
How long do Meta robots.txt changes take?
Meta says its crawlers may cache robots.txt for up to 24 hours.
Sources
Facts on this page were checked against these sources in November 2026. AI crawlers change often, so we review this page regularly.
Check another site, or a competitor's.
See whether Meta AI and every other AI crawler can read any website. Free, with no sign-up.
Run the Meta AI check