TLDR
A Cloudflare and OpenAI research pilot used network-level data from 1% of Cloudflare's infrastructure to surface hundreds of millions of previously unindexed pages for ChatGPT Search. Cloudflare's AI Crawl Control, available on every plan, lets site owners block OAI-SearchBot to stay out of ChatGPT Search independently of GPTBot, which governs model training.
KEY TAKEAWAYS
How a 1% sample found hundreds of millions of missing pages
On 8 July 2026, Cloudflare and OpenAI announced a research pilot to test whether network-level data could make AI search crawling faster and more complete.[1] The results, published on 15 October 2026, landed on a number worth sitting with: using roughly 1% of Cloudflare's network, OpenAI discovered and indexed hundreds of millions of web pages that ChatGPT Search had never seen before.[2]
Cloudflare sits in front of more than one-fifth of the web. Its network generates real-time data on which pages are changing, which URLs are dead and which content is pulling actual traffic, covering an enormous share of the internet as it happens.[1] The pilot gave OpenAI access to exactly that: content freshness, traffic quality and page-level changes, so its crawler could skip dead ends and reach live content it had no other way to locate.[1]
Matthew Prince, Cloudflare's co-founder and chief executive, said the logic plainly: "By sharing our sophisticated network signals, we can find a better way to make AI search more efficient and help people get quality answers faster."[1] Nick Ryder, OpenAI's VP of Research, said the pilot would test "whether network-level insights can help us discover content more efficiently."[1]
Two concrete examples from the Cloudflare blog show what "previously unindexed" looks like in practice: a washing machine user guide and a restaurant's menu and hours. Both matter to someone asking ChatGPT a direct question; neither would have been prioritised by a conventional crawler.[2] One detail from the same post is worth keeping in mind for anyone managing a site: a network-level data point about a page does not grant permission to access it. Discovery and indexing are separate steps, and the usual robots.txt and crawl rules still apply.[2]
The pilot also carries a less obvious cost. Finding new pages can initially increase crawling as those pages are evaluated and indexed, so a site discovered through the pilot may see a temporary spike in bot traffic before things settle.[2]
Your Cloudflare settings and what they control
The pilot's scale makes one practical question urgent for any Australian publisher or brand running behind Cloudflare: what have you told it to allow? Cloudflare's AI Crawl Control, available on every Cloudflare plan, shows which AI services are accessing a site and lets owners allow or block each one.[3]
The distinction that matters most sits between two separate crawlers. OAI-SearchBot is the crawler OpenAI uses for ChatGPT Search, controlling whether your pages appear as sources in ChatGPT answers. GPTBot crawls content for model training, which is a different process entirely. Blocking one through Cloudflare's AI Crawl Control does not automatically block the other.[4] A publisher who blocked GPTBot to keep content out of training data may have assumed they were also absent from ChatGPT Search. That assumption may be wrong.
Cloudflare plans broader access to the capability later in 2026, with no date set, and says it is still working with OpenAI on what better discovery means for business outcomes. Neither company has published how often the newly found pages are cited, or how much traffic they send.[2]
Australian publishers have debated AI citation and referral traffic for months, with little visibility into how crawler decisions shape which sources ChatGPT surfaces and which it skips. The Cloudflare pilot adds another layer: discovery through network data rather than links alone. Crawl permissions still decide access, so the settings worth checking now are the OAI-SearchBot and GPTBot rules in robots.txt and in the Cloudflare dashboard.[4]
SOURCES & CITATIONS
- Cloudflare announces research pilot with OpenAI, Cloudflare press release
- Cloudflare and OpenAI: discovery and the agentic web, Cloudflare blog
- AI Crawl Control, Cloudflare developer documentation
- Manage AI crawlers, Cloudflare developer documentation
- ChatGPT Search can draw on more pages after a Cloudflare pilot, Search Engine Journal
FREQUENTLY ASKED QUESTIONS
What did the Cloudflare and OpenAI pilot actually do?
Does being discovered by the pilot mean my site will appear in ChatGPT answers?
What is the difference between OAI-SearchBot and GPTBot?
How do I check or change my Cloudflare AI crawler settings?
James Dore is founder and editor of Prompt the Market and Strategy Director at Bushnote, a Sydney strategy, SEO and AI search agency. He has a background in strategy, media and financial services, and led consumer product marketing at MONEYME. He writes on AI search, answer engines, technology policy and media economics.
