Do AI Assistants Respect Robots.txt? What a Secret-URL Test Can Actually Prove

A Reddit experiment reported 59 different IP addresses across six Grok prompts, with many requests presenting as ordinary Chrome or Safari browsers. That is a striking lead, but not final proof of a single crawler’s identity. For website owners asking do AI assistants respect robots.txt, the practical payoff is knowing what a private-URL test can demonstrate, what it cannot, and which controls actually protect pages.

The experiment gave each assistant an unlinked, unique URL containing a code, placed a Disallow rule in robots.txt, and logged visits through a Cloudflare Worker. It is a useful design because a correctly isolated URL greatly narrows the likely source of a request. But a request’s user agent alone is not evidence of vendor attribution: Google explicitly warns that user-agent strings can be spoofed. (reddit.com)

What the secret-URL experiment actually showed

The Reddit author created separate hidden URLs for assistants including Claude, ChatGPT, Gemini, Grok, and Perplexity, then asked each fresh chat to open its assigned URL and report a page-specific code. The URLs were reportedly never linked internally, added to a sitemap, or published elsewhere. That setup is far better than searching ordinary access logs, where Googlebot, a browser extension, a cache warmer, a security scanner, or a human visitor could all muddy the result.

The reported observations were mixed:

• Claude allegedly requested robots.txt before accessing a new site and refused the disallowed page.
• ChatGPT allegedly accessed a robots-disallowed page on its first encounter, then an OAI-SearchBot request appeared about 76 seconds later; subsequent attempts were refused.
• Gemini allegedly made requests with a bare Google user agent and did not request robots.txt during the small test.
• Grok allegedly caused bursts from 9 to 12 IPs per request, totaling 59 IPs across six requests, using Chrome- or Safari-like identities.
• Perplexity allegedly visited the root and robots.txt, then warned that the private link might be phishing.

Those are observations from one individual’s one-day test, not a vendor audit or a statistically representative measurement. The most defensible conclusion is narrower: a secret URL plus timestamped origin logs can show that *some retrieval path* accessed a page after an assistant prompt. It does not automatically prove which vendor, model, tool, proxy supplier, or browser automation environment made every request. (reddit.com)

Do AI assistants respect robots.txt? The accurate answer

Some automated AI crawlers do; some user-triggered retrieval systems may not; and a robots.txt rule never physically prevents an HTTP request. Treating “AI” as one crawler category hides the difference that matters.

For example, Google says its common crawlers, including Googlebot and Google-Extended, always obey robots.txt during automatic crawling. OpenAI separately documents OAI-SearchBot for ChatGPT search and GPTBot for model-training related crawling, both of which website owners can manage through robots directives. (developers.google.com)

But Google also states that user-triggered fetchers generally ignore robots.txt because a user initiated the retrieval. That distinction helps explain why a test may show both compliant crawling behavior and a separate, user-requested fetch that reaches a robots-disallowed URL. It also means a test prompt such as “open this link and tell me the code” is testing more than a vendor’s search crawler policy; it may invoke browsing, proxying, safety scanning, browser automation, or a third-party retrieval service. (developers.google.com)

So the useful answer is not “yes” or “no.” It is: identify the exact access mode, then test that mode with verifiable server-side evidence.

Why crawlers, browser agents, and search retrieval must be separated

One page can be reached through at least four operational paths, each with different robots behavior and evidence standards.

1. Automatic search or training crawlers

These systems discover pages independently through links, sitemaps, feeds, or previously known URLs. Google’s common crawlers are the clearest documented example: they use published IP ranges, recognizable reverse-DNS patterns, and obey robots.txt for automatic crawling. A Chrome-like string in the user agent does not make a request human; Googlebot itself can include Chrome-compatible tokens. (developers.google.com)

2. User-triggered fetchers

A user submits a URL, uploads it, asks an assistant to summarize it, or invokes a product feature that retrieves it. Google explicitly classifies these as user-triggered fetchers and says they generally ignore robots.txt. That is not the same thing as a crawler secretly discovering a blocked page. (developers.google.com)

3. Browser automation and computer-use agents

A browser-capable agent may load pages much like a human browser. Its traffic can include JavaScript, image requests, redirects, cookies, and a mainstream browser user agent. If it uses shared cloud infrastructure or a residential proxy network, conventional bot rules based solely on IP or user agent become much less reliable.

4. Third-party sources and cached knowledge

An assistant can sometimes answer correctly without fetching the target URL at all. It could rely on a search index, a cache, a prior crawl, a user-provided page extract, a connected service, or a hallucinated answer that coincidentally matches. This is why a test must use a unique, high-entropy token and server logs—not just ask whether the assistant “knows” a secret.

How to run a repeatable robots-disallowed page test

The original test’s biggest strength was assigning a unique URL to each assistant. We would make the protocol stricter so that another SEO, developer, or agency can reproduce it and distinguish a real fetch from a plausible story.

Build pages that cannot be guessed or discovered

Create one hostname or path per tested system, such as /lab/a7f3-chat/ and /lab/k9p2-grok/. Put a random token of at least 24 characters in visible HTML, in a response header, and in a small text file loaded only by that page. Do not visit the pages first in your own browser—the Reddit author caught this exact mistake after their own browser produced the only logs. (reddit.com)

Use all of the following controls:

1. Exclude every test URL from XML sitemaps, navigation, canonical references, RSS feeds, analytics annotations, and public source repositories.
2. Return Cache-Control: no-store, private and use a different token for every run, so a prior cache cannot satisfy a later prompt.
3. Add X-Robots-Tag: noindex, nofollow in addition to the robots.txt test rule. This does not stop a fetch, but reduces accidental indexing if another crawler finds the URL.
4. Log the exact UTC timestamp, method, URL, status, IP, ASN, complete user agent, referer, request headers, TLS or edge request ID, and response size.
5. Run a control prompt containing a nonexistent URL. If a vendor or intermediary requests it anyway, that reveals prefetching, typo correction, or a different workflow.

Test both allowed and disallowed conditions

Use a direct rule for the relevant token and an all-bots control:

User-agent: *
Disallow: /lab/

User-agent: Googlebot
Disallow: /lab/

User-agent: OAI-SearchBot
Disallow: /lab/

Then repeat on a separate hostname where /lab/ is allowed. The comparison matters. A no-request result on the blocked version could mean robots compliance, a safety refusal, a link-validation failure, an outage, or simply that the assistant never attempted retrieval.

Why Grok’s 59 Chrome-like IPs are an attribution problem

The Grok observation is worth investigating precisely because 59 non-reused IPs over six tests is unusual enough to generate a hypothesis, not a verdict. The reported IPs included hosting providers, small ISPs in Brazil, Mexico, and Poland, and networks associated with Oxylabs. That may suggest proxy rotation or browser-based retrieval, but it does not by itself establish xAI ownership or intent. (reddit.com)

A Chrome or Safari user agent proves almost nothing in isolation. Any script can send it, and legitimate crawlers may also use browser-compatible strings. Google’s documentation explicitly cautions against trusting user agents and provides a verification process instead: reverse-resolve the IP, ensure the hostname is in an approved Google domain, then forward-resolve that hostname back to the original IP. (developers.google.com)

For unidentified AI-related traffic, use a more cautious label in your incident notes:

• Observed: “A request to the Grok-only secret URL occurred 3 seconds after a Grok prompt.”
• Observed: “The request used a Chrome-like user agent and originated from ASN X.”
• Not proven: “Grok itself used this IP.”
• Hypothesis: “The request may have originated from a proxy, browser agent, safety scanner, or vendor-contracted retrieval path.”

That wording preserves the evidence and avoids turning a log pattern into an allegation that cannot be independently verified.

What being blocked by robots.txt actually means

Robots.txt implements the Robots Exclusion Protocol (REP), standardized as RFC 9309 in September 2022. It tells compliant automated clients which URI paths they should avoid fetching. It does not authenticate visitors, hide the URL, encrypt the page, or make the origin server deny a request. (rfc-editor.org)

The RFC’s security guidance is direct: when access must be controlled, operators should use an application-layer security measure such as HTTP authentication. Google similarly warns website owners not to use robots.txt for private content because disallowed URLs can still be exposed and the robots file itself is publicly visible. (rfc-editor.org)

That produces three practical outcomes:

• A compliant crawler may decline to fetch a URL after reading the rule.
• A non-compliant crawler, a browser, or a user-requested fetch can still request it unless your server blocks access.
• Search visibility is a separate issue from crawl permissions. For index control, use noindex where appropriate; for confidentiality, require authorization before content is served.

We see the same distinction in conventional SEO recovery work. If Googlebot is accidentally blocked, robots changes can restore crawl access, but you still need to validate canonical signals, response codes, and index status. Our guide to unblocking Googlebot after a robots.txt mistake walks through that recovery sequence.

Can site owners legally block or bypass AI crawlers with robots.txt?

Website owners can publish robots.txt directives and can deny requests technically through authentication, WAF rules, rate limits, IP controls, and application logic. Whether a particular crawler must comply with a robots instruction under contract, platform terms, copyright law, anti-circumvention rules, computer-access laws, or another legal theory depends on jurisdiction, facts, and the parties involved. Robots.txt itself is a technical protocol, not a legal access-control gate.

The safer operational position is straightforward: if content must not be publicly retrievable, do not rely on an unlinked URL or a Disallow rule. Return a 401 or 403 response before content is delivered, place sensitive resources behind login, and avoid exposing secrets in paths, JavaScript bundles, page titles, or error messages.

For ordinary public marketing content, decide separately whether you want:

• discovery by Google and Bing;
• inclusion in AI search answers;
• training-related crawling where a provider offers a control;
• direct user-requested retrieval; and
• protection from abusive automation.

Those choices rarely map neatly to one User-agent: * rule.

A practical control stack for AI bots ignoring robots.txt

Use robots.txt as a published preference and crawl-management layer, not as your only defense. For a public site with sensitive routes, we recommend a five-layer approach.

1. Authentication first. Put customer portals, previews, exports, staging tools, and internal documentation behind login. A correctly configured 401 challenge is stronger than an obscure URL.
2. Authorization at the application layer. Confirm that a logged-out visitor cannot receive protected HTML, JSON, files, or API responses merely by knowing a path.
3. WAF and rate limits. Challenge or block high-volume suspicious traffic using behavior: request bursts, path enumeration, impossible navigation sequences, datacenter reputation, and session anomalies. Avoid blanket blocks of browser strings, which can block real people.
4. Verified bot allowlists. When allowing Googlebot or another documented crawler, validate IP ownership and reverse DNS rather than trusting the user agent. Google publishes both IP and verification guidance for this purpose. (developers.google.com)
5. Robots and metadata. Maintain clear rules for cooperative bots, then use noindex for URLs you do not want indexed. Keep the two jobs distinct.

For SEO pages you *do* want crawled, a valid sitemap and timely URL notifications remain useful. IndexNow is for telling participating search engines that a URL changed; it does not grant access, override a robots block, or force indexing. That distinction is central to our comparison of Google Indexing API vs IndexNow for bulk URLs.

Turn the test into useful SEO operations

AI crawler investigations can distract from a more urgent issue: accidentally preventing legitimate search engines from finding your new or updated URLs. Keep a separate monitoring process for crawler controls and indexation.

For every important URL, record four states: sitemap inclusion, robots accessibility, canonical target, and index status. A page can be in a sitemap yet blocked; crawlable yet canonicalized away; or submitted through IndexNow yet not indexed because search engines still assess quality and eligibility.

That is where a local workflow helps. Indexa watches XML sitemaps for additions and meaningful updates, then submits eligible URLs to Google’s Indexing API and IndexNow-supported search engines through their official APIs. It is not an AI-crawler firewall, and it cannot compel indexation. Its role is narrower: help us avoid missing the moment a crawlable, canonical URL becomes ready to be discovered.

If pages are visible in a sitemap but remain unprocessed, first work through Google Search Console indexing delays. If the deeper goal is broad, durable search visibility, see what actually works to get pages indexed on Google. Neither workflow should depend on hiding pages through robots.txt.

FAQ

Do AI assistants actually respect robots.txt?

Some do in their automatic crawler modes, but the answer changes by product and access mode. Google says its common crawlers obey robots.txt during automatic crawling, while its user-triggered fetchers generally ignore it. OpenAI documents separate bots and robots controls for automated uses. Treat a user-prompted URL fetch as different from autonomous search crawling. (developers.openai.com)

Which AI bots ignored the robots.txt-disallowed secret URLs?

The Reddit experiment reported that ChatGPT’s initial visit, Gemini, and Grok accessed the disallowed private URLs, while Claude refused and Perplexity did not fetch the target page. Those results are self-reported from a small, one-day test, so they should be treated as leads for repeat testing rather than universal vendor behavior. (reddit.com)

Why did Grok appear from 59 IP addresses using Chrome-like user agents?

The experiment observed 59 IPs across six Grok-related prompts, but this does not conclusively attribute each request to Grok. It could reflect browser automation, rotating proxies, safety scanning, third-party infrastructure, or spoofed traffic. A Chrome-like user agent is weak evidence because user agents are easily imitated; correlate timing, unique URLs, ASNs, headers, and verified vendor infrastructure. (reddit.com)

What does being blocked by robots.txt actually mean?

It means you have published a crawl instruction for compliant automated clients. It does not stop a browser, prove that a vendor will refuse a user-triggered request, or secure confidential content. RFC 9309 recommends real application-layer security such as HTTP authentication when access control is required. (rfc-editor.org)

Is robots.txt still effective for controlling AI crawling?

Yes, as a policy signal for identifiable, compliant automated crawlers—and it remains important for Google’s standard crawling systems. It is not sufficient for protecting public URLs against every AI browser, scraper, or proxy-mediated fetch. Pair it with authentication, server-side authorization, rate limits, and verified bot identification whenever access truly matters. (developers.google.com)