All of Cloudflare’s new customers will now be asked if they want to allow AI companies’ bots to scrape their site, and for what purpose. Cloudflare’s scale gives it a better chance than most of enabling something like a collective response by content sites that want to force AI firms to cough up. It is testing a pay-as-you-crawl system that would let sites charge bots an entry fee. “We have to set the rules of the road,” says Mr Prince, who says his preferred outcome is “a world where humans get content for free, and bots pay a tonne for it”.