Cloudflare activated a sweeping change to the way AI companies access the web. Every new domain that signs up to its network now blocks unverified AI crawlers by default. This affects an estimated 20% of the entire Internet.
Site owners are prompted up-front to decide whether to allow bot access. They can also charge for it or flat-out refuse it. Cloudflare CEO Matthew Prince framed this as “put[ting] the power back in the hands of creators”.
Beyond “Allow” or “Deny”: meet HTTP 402
Cloudflare’s parallel launch of Pay Per Crawl lets publishers price each bot request. When an AI agent hits a pay-walled URL, it receives an HTTP 402 Payment Required response. This response advertises the cost. A bot can then retry the request with a signed payment header to complete the transaction. Alternatively, it can back off if the price is too steep.
Cloudflare acts as merchant-of-record and enforces the rule set after its existing WAF and bot-management layers.
Why the clamp-down?
Internal telemetry published on Cloudflare Radar helps explain the urgency. During the week of 19–26 June, Anthropic’s Claude crawler made around 70,900 HTML requests for each referral click-through. Meanwhile, OpenAI’s GPTBot had a ratio that remained in the thousands-to-one range.
Put bluntly: most generative-AI systems slurp content on the web without sending readers or revenue back.
Publishers cheer, but keep one hand on the steering wheel
Media groups were quick to endorse the move. DMGT vice-chair Rich Caccappolo called the default block “a structured and transparent relationship between content creators and AI platforms”.
News/Media Alliance president Danielle Coffey likewise praised the framework for creating “a more equitable exchange” between the two industries.
AI crawlers are still growing fast
Even as Cloudflare rolls out stricter controls, overall crawler traffic continues to swell. Company data show a year-over-year rise of 18% in combined AI-and-search crawling. GPTBot volume increased by 305% in the same period. Googlebot also grew by 96% as Google leans into AI Overviews.
Developers react: opportunity ≠ headlock
A lively Hacker News thread shows developers split: some hail Pay Per Crawl as a long-needed revenue lane, while independent builders fear roadblocks for open-source search projects or archival crawls. Some have expressed concerns that this scheme could tax innovation before it starts.
What happens next?
- For publishers: The default block is live today for new Cloudflare domains; existing customers can opt-in from the dashboard.
- For AI firms: Authenticating crawlers, and perhaps negotiating per-URL pricing, becomes table stakes for large-scale training.
- For regulators: Cloudflare’s protocol could crystallize around its proposed Web Bot Auth standard, giving lawmakers a technical hook for future policy.
Whether the market embraces HTTP 402 micro-payments or gravitates toward negotiated bulk licenses, Cloudflare has lobbed a high-profile gauntlet: “scrape-first, apologize-later” is no longer the default on one-fifth of the web.
