Is Your Website Blocking AI Crawlers? How to Check and Fix It

A company publishes strong content, improves its structured data, builds out its service pages—and still does not appear in AI-generated search results. It is easy to assume the content is the problem. Often, it is not.

The platform may simply be unable to retrieve the page.

Access problems can happen at several layers at once: robots.txt, page-level directives, hosting, a CDN, a firewall, a security plugin, an authentication wall, a server response, how the page renders, a geographic restriction, or an indexing setting. Any one of them can quietly make your content invisible to the systems you are trying to reach.

Crawler access is the first requirement, not the whole strategy. This article is the access-layer companion to the rest of our AI Search Visibility series: if you have already read why your business is not appearing in AI search results, this is how to diagnose and fix the technical half of that problem.

What is an AI crawler?

An AI crawler is an automated program that requests pages from your website on behalf of an AI company. That is where the simple part ends, because different crawlers exist for very different reasons. Depending on the program, an AI crawler may:

• Discover public web content
• Retrieve pages to support search answers
• Support citations and links back to your site
• Collect data used to train models
• Refresh previously indexed information
• Fetch a single page in response to a specific user request

“AI crawler” is a broad label, not one universal technology. Treating every AI bot as interchangeable is the first mistake—because the crawler that decides whether you appear in ChatGPT’s search results is not the same as the crawler that gathers training data.

Search crawlers and training crawlers are not the same

Before you change a single line of robots.txt, it helps to understand the three broad categories a bot can fall into. Blocking one does not necessarily block the others, and confusing them is how businesses accidentally remove themselves from AI search while trying to opt out of training—or leave themselves exposed while trying to appear.

OpenAI

OpenAI documents separate, independently controllable user-agents. OAI-SearchBot is “used to surface websites in search results in ChatGPT’s search features”—this is the one to allow if you want to appear in ChatGPT. GPTBot crawls content that may be used to train models. ChatGPT-User handles user-triggered fetches when a person’s request causes ChatGPT to visit a page. Because each is independent, blocking GPTBot is not the same as blocking OAI-SearchBot—you can opt out of training while remaining eligible for ChatGPT search. Allowing any of them still does not guarantee you will be cited or recommended.

Perplexity

Perplexity documents PerplexityBot, described as designed to surface and link websites in Perplexity’s search results (not to collect training data), and Perplexity-User, which visits a page to help answer a specific user question and includes a link in the response. PerplexityBot does not represent every Perplexity request, so a site’s own analytics will not reveal every unclicked mention.

Google

Google does not use a separate public AI crawler for AI Overviews or AI Mode. Those features are built into Search and draw on Google’s existing index, so ordinary Googlebot access, indexability, and snippet eligibility are what matter—there is no special crawler rule or schema to “turn on.” The separate Google-Extended token is a control for training and grounding in Google’s other generative systems (such as Gemini); it does not govern whether you appear in Search or AI Overviews. In other words, blocking Google-Extended does not remove you from AI Overviews, and there is no AI-Overviews-specific crawler to allow. For the visibility side of this, see how to show up in Google AI Mode.

Other platforms

Bing and other engines have their own crawlers (for example, Bingbot), and Bing’s index also feeds some third-party AI experiences. The principle is the same everywhere: identify the crawler that supports the search product you care about, confirm what its official documentation says, and don’t assume one bot name covers every product a company operates.

A crawler reference table

The same information, at a glance. Verify names and behavior against each platform’s current documentation before you act on them—crawler policies change.

The seven layers that can block AI access

robots.txt gets all the attention, but it is only the first of seven layers where access can break. A request has to survive all of them to become usable content.

1. robots.txt

robots.txt lives at the root of your domain (/robots.txt) and tells compliant crawlers which paths they may request. Rules are grouped by User-agent, with Disallow and Allow paths and wildcard support. Ordering and grouping matter, and duplicate broad rules are a common source of confusion—a stray global block can override the specific permissions you intended.

The most damaging pattern is the one left over from a staging site:

To opt out of training while staying eligible for AI search, you control each user-agent independently. This is illustrative—confirm each token against current documentation, and remember that adding a crawler’s name does not by itself improve visibility:

Two cautions. First, robots.txt controls crawling, not indexing—disallowing a URL does not remove a page that is already indexed, and blocking crawling can prevent systems from ever seeing your updates. Second, an llms.txt file is not required for AI visibility; it is an emerging proposal, not a rule any major platform mandates.

2. Page-level directives

Even when crawling is allowed, a page can be held back by its own instructions: a noindex robots meta tag or X-Robots-Tag HTTP header removes it from search eligibility, a nofollow can limit discovery, and a canonical tag pointing elsewhere tells engines to prefer a different URL. A single template-wide noindex left on after a redesign can quietly de-index an entire section. Check the rendered <head> and the response headers, not just the source you expect.

3. Server responses and stability

A crawler only gets what your server returns. A 403 Forbidden or 401 refuses access; a 404/410 says the page is gone; a 429 Too Many Requests or repeated 5xx errors and timeouts make a page unreliable to fetch. Crucially, a 200 OK proves the request succeeded—not that the full page rendered or that the content a crawler needs is present. Intermittent server instability is especially corrosive because it fails silently and inconsistently.

4. CDN and firewall rules

This is the layer that most often surprises people. A robots.txt test can pass while a CDN or web application firewall (WAF) blocks the same crawler by IP range, user-agent, rate, or “suspicious traffic” heuristics—before the request ever reaches your origin. A successful robots.txt check does not prove the CDN or firewall allows the request. Review your CDN’s bot rules and firewall event logs for legitimate crawlers being challenged or denied.

5. Bot management, security plugins, and rate limiting

Bot-mitigation services, aggressive security plugins, and rate limits can throttle or challenge crawlers with JavaScript checks or CAPTCHAs that automated fetchers cannot solve. These tools exist for good reasons, and the answer is never to disable security globally. The answer is to distinguish legitimate, documented search crawlers from abusive traffic and to allow the former deliberately—while still blocking private, admin, and sensitive areas.

6. JavaScript rendering

Many crawlers request your HTML but do not execute client-side JavaScript the way a browser does. If your core content, links, or key facts only appear after JS runs, a crawler may receive a nearly empty shell and a 200 OK at the same time. Server-render or statically render the content that matters—headings, body copy, and internal links—so it is present in the initial HTML response. This is often a website development fix as much as an SEO one.

7. Access walls: login, geography, and staging

Some content is hidden by design, and sometimes by accident. Login walls and paywalls put content behind authentication—and crawlers should not be given a way around authentication or private content. Geographic restrictions can serve a block or a different page to requests from certain regions. And the classic accident is a staging or development environment whose Disallow: /, noindex, or HTTP-auth protection is copied to production during a migration. After any launch or migration, confirm production is open and staging is not indexed.

How to check whether your site is blocking AI crawlers

Test the pages that earn business—service pages, key articles, location pages, product pages—not just the homepage. Work down the stack:

Server log files are the ground truth: they show which crawlers actually reached your site, what status they received, and whether a bot you expected never appeared at all. Google Search Console’s crawl and indexing reports add the Google-specific view. Together they turn “we think we’re accessible” into evidence.

Common access mistakes

• Testing only the homepage and assuming the whole site is accessible
• Trusting a passing robots.txt test while a CDN or firewall silently blocks the crawler
• Reading a 200 OK as proof the content rendered
• Blocking GPTBot to opt out of training and assuming that removes you from ChatGPT search (it doesn’t—that’s OAI-SearchBot)
• Looking for a Google “AI Overviews crawler” that does not exist
• Leaving a staging Disallow: / or noindex live in production after a migration
• Applying a template-wide noindex and de-indexing a whole section by accident
• Allowing every unknown bot in the name of “AI visibility”
• Disabling security globally instead of allowing specific documented crawlers
• Adding crawler names to robots.txt and expecting visibility to improve on its own
• Never re-testing access after hosting, CDN, plugin, or firewall changes

How BuckStone audits and fixes access

Access problems rarely live in one place, which is why they resist single-tool fixes. BuckStone works across the whole stack:

Crawl & indexing layer

We audit robots.txt group by group, page-level directives, canonicals, and X-Robots-Tag headers, and reconcile them with Google Search Console’s crawl and index reports—part of our broader technical SEO work.

Hosting, CDN & firewall layer

We review server responses and stability, CDN bot rules, and WAF and security-plugin logs to find legitimate crawlers being challenged, throttled, or denied—then adjust the rules precisely rather than bluntly.

Rendering layer

We confirm that the content that matters is present in the initial HTML response, and fix client-side-only rendering at the development level where needed.

Evidence & monitoring

We use server logs and Search Console to prove which crawlers actually arrive and what they receive, and we re-test after migrations and infrastructure changes. Access is where our five-part framework starts—and once a site is reliably accessible, the work continues into understanding, evidence, corroboration, and measurement. Fixing access does not guarantee citations or recommendations; it removes the barrier that makes them impossible.

Final answer: is your website blocking AI crawlers?

Maybe—and the only way to know is to test the full access stack on your most important pages. Confirm robots.txt allows the search crawlers you want, check page-level directives and canonicals, verify server responses and stability, review CDN and firewall behavior, make sure your content renders without JavaScript, and rule out login, geographic, or staging restrictions. Use server logs and Search Console as your evidence.

Then keep it in balance: open to documented search crawlers on public pages, deliberate about training crawlers, and still closed to private data and abuse. Access is necessary, not sufficient—but until it is solved, nothing downstream can work. Once it is, the next question is how AI systems understand your business. If you would rather have it audited and fixed across the whole stack, that is the work we do.

Frequently asked questions