Cloudflare OpenAI Pilot: What It Means for AI Search Discovery

Cloudflare says a research cohort covering approximately 1% of its network helped OpenAI identify hundreds of millions of URLs that had not previously been known to OpenAI’s search systems. The Cloudflare OpenAI pilot is useful news for publishers because it highlights another possible AI search discovery input—and gives us a practical reason to improve sitemap accuracy, internal linking, and change notifications.

As of October 18, 2026, the public findings do not reveal ChatGPT Search’s ranking algorithm or promise that every discovered URL was indexed, cited, or sent traffic. The payoff for site owners is a clearer channel-aware workflow: make important pages easy to discover, technically available to crawl, and accurately described when they change.

What the Cloudflare OpenAI pilot says—and what it does not

Cloudflare’s pilot announcement describes a joint research effort with OpenAI. Cloudflare says it supplied real-time network signals from a participating cohort of roughly 1% of its network to help OpenAI find pages that conventional discovery methods had not surfaced.

The announcement reports that OpenAI identified hundreds of millions of previously unknown pages through the work. That is a substantial discovery result. It is not the same claim as “hundreds of millions of pages entered ChatGPT Search’s index,” and Cloudflare’s wording matters: the signals can help OpenAI determine which pages are worth crawling, indexing, or updating.

We can separate the findings into two categories.

What Cloudflare reports:

• A limited pilot used real-time network signals from about 1% of Cloudflare’s network.
• Those signals helped OpenAI discover a very large number of URLs it had not previously known.
• The purpose was to improve how OpenAI identifies content worth crawling, indexing, or refreshing.

What the public findings do not establish:

• That every discovered URL was crawled or indexed.
• That Cloudflare submits individual customer URLs directly to ChatGPT Search.
• That links, XML sitemaps, or conventional crawlers are no longer useful.
• That a discovered page will rank, be cited in an answer, or receive referral visits.
• That publishers can access a public OpenAI URL-submission API.

That boundary is our interpretation of the published material, not a criticism of the pilot. Discovery is an early stage in search; improving it can be meaningful even without public visibility into the rest of OpenAI’s retrieval pipeline.

Discovery, crawling, indexing, and ranking are separate stages

The most useful way to read the Cloudflare announcement is to distinguish four jobs that are often collapsed into the word “indexing.” Google’s documentation makes similar distinctions when explaining sitemaps and crawlable links.

1. Discovery: A system learns that a URL exists. Links, XML sitemaps, feeds, and infrastructure-level signals can all contribute.
2. Crawling: A bot requests the URL and retrieves available content. A discovered URL can still fail to load, redirect, return an error, or be inaccessible under a publisher’s rules.
3. Indexing: The system processes the fetched content and may store it for later retrieval. Crawling does not guarantee indexing.
4. Ranking or retrieval: The search product decides whether to use that indexed material for a query, result, or AI-generated answer.

Cloudflare’s published findings concern improved awareness and prioritization: signals can tell OpenAI that a URL may exist, may have changed, or may merit another look. They do not disclose how ChatGPT Search weighs relevance, source quality, links, freshness, or any other ranking factor.

Google gives publishers a comparable warning for a familiar channel: submitting a sitemap helps Google discover URLs, but does not guarantee crawling, indexing, or ranking. We recommend applying that same discipline to AI-search claims. A new discovery source is valuable, but it is not a visibility guarantee.

Why network signals may find pages links and sitemaps miss

Cloudflare operates network infrastructure used by websites, which puts it in a different position from a crawler that works outward from known URLs. According to Cloudflare, real-time network signals can surface useful pages that may otherwise be hard for an AI search system to notice promptly.

A realistic example is a retailer that publishes 2,000 region-specific product pages in a weekend. The pages may be public and valid, but perhaps a sitemap-generation task runs only nightly, category links are added later, and some variants have little external-link visibility. A crawler relying solely on older URL inventories and links may not prioritize every new page quickly.

That example is general technical SEO reasoning from us, not a disclosed Cloudflare pilot use case. The pilot does not publish the complete set of signals, their weighting, or URL-level outcomes. We therefore should not assume that a request, page view, cache event, or other network activity automatically becomes an OpenAI indexing request.

Network signals also cannot repair a technically poor page. A URL that serves an error, has contradictory canonical instructions, contains little accessible content, or is unavailable to a crawler can still fail at later stages. The sensible conclusion is not to hide pages from ordinary crawl paths; it is to provide several consistent paths to the pages that matter.

Does the pilot mean ChatGPT Search needs links less than Google?

No public evidence from the pilot supports that broad conclusion. Cloudflare’s results suggest that links are not the only possible route for OpenAI to learn about a page. That is different from proving that links have little value in ChatGPT Search ranking or that OpenAI uses them less than Google does.

Google explains that crawlable links help Google find pages and understand site relationships. Internal links also help human visitors move between a hub page, category, product, article, and supporting resource. Those benefits remain useful regardless of how any one AI system discovers URLs.

For publishers, we recommend maintaining three complementary discovery paths:

• Crawlable internal links to important pages from relevant category, hub, or related-content pages.
• An XML sitemap containing canonical URLs the publisher wants crawled and potentially indexed.
• Timely, engine-specific notifications where an official protocol accepts them.

This approach is not derived solely from the Cloudflare OpenAI pilot. It combines Cloudflare’s discovery finding with established Google guidance and ordinary site-maintenance practice. For a Google-focused comparison of manual and automated URL handling, see our guide to indexing a URL with Google Search Console versus Indexa.

Cloudflare is not a public ChatGPT Search submission service

The Cloudflare announcement describes a research collaboration, not a dashboard or endpoint where a publisher can submit a URL to ChatGPT Search. Cloudflare provides discovery signals within the pilot; OpenAI decides what to crawl, process, update, and potentially use in its products.

OpenAI’s own crawler documentation is the appropriate source for publisher controls relating to OpenAI bots. It describes separate crawlers and their purposes, including search-related crawling and training-related crawling. Those controls concern crawler access; they are not a promise of indexing or ranking.

For that reason, we would not describe Cloudflare as an OpenAI indexing tool. Nor should a site owner assume that using Cloudflare causes every new sitemap URL to be sent directly to OpenAI. The public evidence supports a narrower statement: participating network signals may improve OpenAI’s ability to identify pages worth evaluating.

Our recommendation is to treat ChatGPT Search as a discovery and referral channel that benefits from accessible public content, while avoiding unsupported promises about inclusion. If OpenAI publishes a general URL-notification interface in future, that would be a separate mechanism requiring its own documentation and workflow.

XML sitemaps remain the publisher-controlled URL inventory

The strongest publisher-controlled discovery asset is still an accurate XML sitemap. Google recommends sitemaps especially for large, new, complex, or media-heavy sites, while also making clear that links remain useful for discovery.

The sitemap protocol limits one sitemap file to 50,000 URLs or 50 MB uncompressed. Larger sites can use multiple sitemap files and a sitemap index. An agency might separate article, product, location, image, and document sitemaps so errors and changes are easier to isolate.

A useful sitemap should contain URLs that are genuinely intended to be found. We recommend excluding:

• URLs returning redirects, 404s, or server errors.
• Parameter variants and duplicate URLs that canonicalize elsewhere.
• noindex pages, staging environments, and internal search-result pages that are not intended for search.
• Retired products or expired pages with no continuing search value.

The lastmod value deserves the same care. Update it after a meaningful editorial, product, availability, or specification change—not merely because a site-wide footer changed. This is our operational recommendation, rather than a finding from the Cloudflare pilot, but it creates a cleaner change signal for crawlers and monitoring tools.

Use notification channels according to their documented scope

There is no universal “index this everywhere” button. Google, IndexNow-supported engines, and ChatGPT Search expose different publisher mechanisms, and each has limitations.

Google: sitemap and link discovery for most pages

For ordinary articles, service pages, product pages, and documentation, Google’s documented baseline is crawlable links and sitemaps. Search Console provides diagnostics and URL inspection, but a request does not compel indexing.

Google’s Indexing API documentation limits the API to pages with JobPosting structured data and livestream pages using BroadcastEvent in a VideoObject. It is not a general-purpose API for ordinary commercial or editorial URLs. That limitation comes from Google, not from the Cloudflare/OpenAI work.

IndexNow: a protocol for participating engines

IndexNow documentation describes a protocol for notifying participating search engines when URLs are added, updated, or deleted. Its documented batch endpoint accepts up to 10,000 URLs per request after key verification. Receipt confirms the notification was received; it does not guarantee crawling, indexing, or ranking.

IndexNow is a separate protocol from Google’s Indexing API and from Cloudflare’s OpenAI pilot. Before building a workflow around it, verify the current participating engines and their implementation guidance in IndexNow’s own documentation.

A practical publishing workflow after the Cloudflare OpenAI pilot

Our recommended workflow is deliberately conservative. It does not assume hidden access to ChatGPT Search, and it does not convert general technical SEO advice into a claim about the pilot.

When publishing a page

Check that the final canonical URL returns a successful response, uses the intended canonical annotation, and is not accidentally marked noindex. Link to it from a relevant page that people and crawlers can reach. Add it to the appropriate XML sitemap if it is an important indexable URL.

For a downloadable guide or other document, confirm the asset URL is public, stable, and included in the relevant sitemap where appropriate. Static assets can have their own discovery issues; our guide to indexing a PDF in Google covers those checks in a Google-specific context.

When updating or removing a page

After a material update, revise the sitemap’s lastmod value and record the URL change. Send a notification through IndexNow where it is supported. If a URL falls into Google’s narrow documented Indexing API use cases, use that API; otherwise rely on Google’s normal crawling, sitemap, and Search Console processes.

For a removed page, delete it from the sitemap. Use a 301 redirect only when a genuinely relevant replacement exists; otherwise a 404 or 410 can be the honest response. These are general publishing recommendations, not claimed effects of the Cloudflare research.

Where Indexa fits—and where it does not

Indexa is a local desktop utility for monitoring XML sitemaps and detecting new or updated URLs. Its role is operational: help site owners and agencies notice sitemap changes and send notifications through supported official channels, including IndexNow and Google workflows where the URL and API use case are eligible.

Indexa does not submit URLs to ChatGPT Search, create a private connection to Cloudflare’s pilot, or guarantee Google indexing, AI citations, rankings, or traffic. It cannot make an ineligible ordinary blog post eligible for Google’s restricted Indexing API.

For a practical use case, an agency can monitor separate client sitemaps, keep a record of detected URL changes, and notify IndexNow-supported engines after new or changed pages are published. The sitemap remains the source of truth; the notification is an additional prompt, not proof of inclusion. That distinction also matters when comparing Google API quotas and indexing tools.

What publishers should measure

The “hundreds of millions” figure is an aggregate finding from Cloudflare’s pilot, not a forecast for an individual domain. We recommend measuring controllable operational outcomes rather than promising clients a specific AI-search traffic result.

A practical monthly report can track:

• Valid canonical URLs in each XML sitemap and URLs excluded because of errors or redirects.
• Time between publishing a new URL and its appearance in the sitemap.
• New and changed URLs detected, plus accepted IndexNow notifications where applicable.
• Google Search Console coverage and crawl diagnostics for Google-specific visibility.
• Referral sessions and conversions from identifiable AI-search traffic, when analytics provides them.

This framework is ours. It separates technical readiness from search-engine decisions and business outcomes. A notification can be accepted without a crawl; a page can be indexed without ranking; and a cited page may produce little traffic. Keeping those events separate prevents both inflated claims and premature pessimism.

FAQ

How did Cloudflare help OpenAI discover hundreds of millions of pages?

Cloudflare says its pilot used real-time network signals from a cohort representing about 1% of its network to help OpenAI identify URLs that were previously unknown to its search systems. The announcement says the signals help OpenAI decide which pages may be worth crawling, indexing, or updating; it does not say every identified page entered a search index.

Does the Cloudflare OpenAI pilot mean ChatGPT Search relies less on links than Google?

Not based on the public findings. The pilot supports the narrower conclusion that OpenAI can use an additional discovery input beyond links and sitemaps. OpenAI has not published a ranking formula or a comparison of link weighting with Google. We recommend retaining strong internal links for navigation, context, and conventional crawler discovery.

What signals can OpenAI use to find pages that sitemaps and links miss?

Cloudflare refers to real-time network signals but does not publicly enumerate every signal or its weighting. We should not infer that private content, every visitor request, or every network event is exposed to OpenAI. The documented result is improved identification of pages worth evaluating, not a public specification for an AI indexing feed.

Does Cloudflare submit URLs directly to ChatGPT Search?

The public announcement does not describe a direct URL-submission product for ChatGPT Search. It describes a research pilot in which Cloudflare supplies discovery signals and OpenAI makes subsequent crawl and indexing decisions. OpenAI’s crawler documentation covers bot access, but it does not provide a general-purpose publisher endpoint for immediate URL submission.

What does Indexa do for AI search discovery?

Indexa monitors XML sitemaps for new and changed URLs and helps route notifications through supported official channels, such as IndexNow and eligible Google workflows. It does not send URLs to ChatGPT Search or Cloudflare’s research pilot, and it cannot guarantee indexing or citations. Its value is reliable change detection and channel-aware notification management.