Cloudflare To Block AI Bots From Scraping Content, Launches Marketplace To Let Websites Charge For Crawling

Cloudflare Inc. (NET), a connectivity cloud company, has rolled out updates aimed at helping publishers assert more control over AI crawlers that scrape online content for training their models.

The web giant now offers tools that allow site owners to automate their robots.txt file and restrict AI bot access specifically to ad-monetized pages. A robots.txt file is a plaintext document that tells web crawlers which areas of a site they’re permitted to visit and which sections they should avoid.

Despite the news, Cloudflare stock inched lower by 0.9% in Tuesday's pre-market trading session.

Cloudflare launched these new tools in response to growing concerns that AI-powered bots are harvesting website content without generating substantial referral traffic to the original publishers.

Historically, digital publishers embraced bots like Googlebot, which helped boost their content’s exposure in search results. However, a new wave of AI-focused crawlers, from companies like OpenAI and Anthropic, have been scraping data online to train their models, often without sending visitor traffic in return. 

Cloudflare’s analysis shows that OpenAI’s GPTBot made roughly 1,700 content requests for each referral it generated, while Anthropic’s ClaudeBot exhibited an even more extreme ratio of 73,000:1.

In response to the changing landscape, Cloudflare launched a managed robots.txt feature that automatically creates or modifies a site's file to signal AI crawlers such as Google-Extended and Applebot-Extended to avoid using the content for training purposes. 

The tool preserves any existing rules and remains SEO-friendly, making it easy for users, even without technical expertise, to express how they want their site data treated.

Cloudflare’s latest feature detects advertising elements in a page’s code, including scripts and HTML markers, and blocks AI crawlers only on those pages.

The company will now prompt all new customers to enable its managed robots.txt by default. Both new tools are free and available to all Cloudflare users. 

On Stocktwits, retail sentiment around shifted to ‘neutral’ from ‘bullish’ territory the previous day, amid ‘high’ message volume levels.

Cloudflare stock has gained over 81% year-to-date and has more than doubled in the last 12 months.

Also See: Palantir Ranks As S&P 500’s Best Performer Mid-Year, But Retail Traders Tread With Caution

For updates and corrections, email newsroom[at]stocktwits[dot]com.