# AI Crawler Statistics 2026: Traffic, Training, Referrals and Blocking

Updated 2026-11-12. Checked by StatisticsGlobal editorial team. 2 research studies and 3 vendor network datasets, 8 sources.
Canonical: https://statisticsglobal.org/ai-crawler-statistics/

52% of crawler requests on Cloudflare's network were for AI training in June 2026, up from 22% in spring 2025, according to Cloudflare. AI crawlers send little traffic back. In July 2025 Anthropic crawled about 38,066 pages per referred visitor and OpenAI about 1,091, against 5.4 for Google. TollBit found European publishers got one human referral per 179 AI scrapes in H1 2026. Blocking is common among news sites: 48% of top news sites in 10 countries blocked OpenAI by the end of 2023.

## Key statistics

- 52% of crawler requests on Cloudflare's network were for AI training in June 2026, up from 22% in spring 2025, according to Cloudflare.
- Anthropic crawled about 38,066 pages for every visitor it referred in July 2025, down from 286,930 in January 2025, according to Cloudflare.
- European publishers received one human referral for every 179 AI bot scrapes in H1 2026, over 3 times worse than North American sites, according to TollBit.
- AI bots other than Googlebot made an average of 4.2% of HTML requests on Cloudflare's network in 2025, according to Cloudflare.
- 48% of top news websites in 10 countries blocked OpenAI's crawlers by the end of 2023, according to the Reuters Institute.

## AI crawler traffic statistics

AI crawlers are a small share of all web requests but a fast-growing share of bot traffic. How big they look depends on whether Googlebot, which crawls for both search and AI, is counted.

AI bots other than Googlebot made 4.2% of HTML requests on Cloudflare's network in 2025. Googlebot alone made 4.5%. Add the two and AI-capable crawlers reached 8.7% of page requests. Cloudflare keeps them apart because Googlebot feeds both Google Search and Google's AI products, and site owners cannot block one use without the other.

Fastly sees a much smaller number. AI bots were 8% of the bot traffic it classes as wanted in January 2026. Wanted bots were only about 1% of all requests, so AI bots made roughly 0.08% of everything. The gap with Cloudflare is mostly definitional. Fastly does not count search engine crawlers as AI bots, and it puts unverified scrapers in its unwanted bot group, which made 48% of requests.

Both networks agree that bots are now close to half of all traffic. Bots made 49% of requests on Fastly in January 2026 and 60% of the requests that reached origin servers. On Cloudflare, humans made 47% of HTML requests on 2 December 2025. For the wider picture of how AI tools send people to websites, see our [AI search statistics](https://statisticsglobal.org/ai-search-statistics/).

- AI bots other than Googlebot made an average of 4.2% of HTML requests on Cloudflare's network in 2025, according to Cloudflare. Source: Cloudflare, The 2025 Cloudflare Radar Year in Review (15 December 2025, accessed 12 November 2026) https://blog.cloudflare.com/radar-2025-year-in-review/
- Googlebot alone made 4.5% of HTML requests on Cloudflare's network in 2025, according to Cloudflare. Source: Cloudflare, The 2025 Cloudflare Radar Year in Review (15 December 2025, accessed 12 November 2026) https://blog.cloudflare.com/radar-2025-year-in-review/
- AI bots and Googlebot together made 8.7% of HTML requests on Cloudflare's network in 2025, calculated by StatisticsGlobal from Cloudflare data. Source: Cloudflare, The 2025 Cloudflare Radar Year in Review (15 December 2025, accessed 12 November 2026) https://blog.cloudflare.com/radar-2025-year-in-review/
- Humans generated 47% of HTML requests on Cloudflare's network on 2 December 2025, and non-AI bots 44%, according to Cloudflare. Source: Cloudflare, The 2025 Cloudflare Radar Year in Review (15 December 2025, accessed 12 November 2026) https://blog.cloudflare.com/radar-2025-year-in-review/
- AI and search crawler traffic grew 18% between May 2024 and May 2025 on a fixed set of Cloudflare customers, according to Cloudflare. Source: Cloudflare, From Googlebot to GPTBot: who's crawling your site in 2025 (1 July 2025, accessed 12 November 2026) https://blog.cloudflare.com/from-googlebot-to-gptbot-whos-crawling-your-site-in-2025/
- Googlebot crawl requests grew 96% between May 2024 and May 2025, according to Cloudflare. Source: Cloudflare, From Googlebot to GPTBot: who's crawling your site in 2025 (1 July 2025, accessed 12 November 2026) https://blog.cloudflare.com/from-googlebot-to-gptbot-whos-crawling-your-site-in-2025/
- AI crawlers generated 20% of verified bot traffic on Cloudflare's network in 2025, half the 40% from search engine crawlers, according to Cloudflare. Source: Cloudflare, The 2025 Cloudflare Radar Year in Review (15 December 2025, accessed 12 November 2026) https://blog.cloudflare.com/radar-2025-year-in-review/
- Bots made 49% of requests to sites on Fastly's network in January 2026, and humans 51%, according to Fastly. Source: Fastly, Fastly Threat Insights Report 26Q1: AI, Bots, and the Agentic Future of the Web (April 2026, accessed 12 November 2026) https://learn.fastly.com/rs/025-XKO-469/images/Fastly-Threat-Insights-Report-26Q1.pdf
- Bots made 60% of requests that reached origin servers on Fastly's network in January 2026, according to Fastly. Source: Fastly, Fastly Threat Insights Report 26Q1: AI, Bots, and the Agentic Future of the Web (April 2026, accessed 12 November 2026) https://learn.fastly.com/rs/025-XKO-469/images/Fastly-Threat-Insights-Report-26Q1.pdf
- AI bots made about 8% of wanted bot traffic on Fastly's network in January 2026, according to Fastly. Source: Fastly, Fastly Threat Insights Report 26Q1: AI, Bots, and the Agentic Future of the Web (April 2026, accessed 12 November 2026) https://learn.fastly.com/rs/025-XKO-469/images/Fastly-Threat-Insights-Report-26Q1.pdf
- AI bots classed as wanted made roughly 0.08% of all requests on Fastly's network in January 2026, calculated by StatisticsGlobal from Fastly data. Source: Fastly, Fastly Threat Insights Report 26Q1: AI, Bots, and the Agentic Future of the Web (April 2026, accessed 12 November 2026) https://learn.fastly.com/rs/025-XKO-469/images/Fastly-Threat-Insights-Report-26Q1.pdf
- AI bots made 4% of wanted bot traffic to publishing sites on Fastly's network in January 2026, according to Fastly. Source: Fastly, Fastly Threat Insights Report 26Q1: AI, Bots, and the Agentic Future of the Web (April 2026, accessed 12 November 2026) https://learn.fastly.com/rs/025-XKO-469/images/Fastly-Threat-Insights-Report-26Q1.pdf

Table: How big AI crawling looks in each network dataset, by measure and period

| Source | Period | Measure | Figure |
|---|---|---|---|
| Cloudflare | 2025 | AI bots excluding Googlebot, share of HTML requests | 4.2% |
| Cloudflare | 2025 | Googlebot, share of HTML requests | 4.5% |
| Cloudflare | 2025 | AI crawlers, share of verified bot traffic | 20% |
| Fastly | January 2026 | AI bots, share of wanted bot traffic | 8% |
| StatisticsGlobal calculation | January 2026 | Wanted AI bots, share of all Fastly requests | about 0.08% |

Sources: Cloudflare 2025 Radar Year in Review, Fastly Threat Insights Report 26Q1. Fastly share of all requests calculated by StatisticsGlobal.

## Most active AI crawlers

Which AI crawler is "biggest" depends on the network and the month. OpenAI's GPTBot led Cloudflare's AI-only ranking in 2025. On Fastly in early 2026, Google, Meta and Huawei crawlers led.

GPTBot's share of AI-only crawling on Cloudflare rose from 5% in May 2024 to 30% in May 2025, a sixfold rise. Bytespider went the other way, from 42% to 7.2%. Meta's crawler arrived at 19%. By July 2025 GPTBot held 28.1% and ClaudeBot 23.3%.

Fastly's January 2026 data ranks the field differently. GoogleOther made 36% of AI crawler traffic, Meta 28% and Huawei's PetalBot 18%, so three bots made 82% of it. GPTBot ranked lower there. The two networks serve different customers, and they classify Google's crawlers differently, so neither ranking is the single "true" one.

Location matters too. GPTBot was the second most active crawler for Asia-based Cloudflare customers in October 2025, at 14.0%, but third in Europe at 10.8%, behind Bingbot. Googlebot led in every region. Retail and computer software sites drew just over 40% of AI crawling.

- GPTBot had a 30% share of AI-only crawler traffic on Cloudflare's network in May 2025, up from 5% in May 2024, according to Cloudflare. Source: Cloudflare, From Googlebot to GPTBot: who's crawling your site in 2025 (1 July 2025, accessed 12 November 2026) https://blog.cloudflare.com/from-googlebot-to-gptbot-whos-crawling-your-site-in-2025/
- GPTBot's share of AI-only crawler traffic rose 25 percentage points, a sixfold increase, between May 2024 and May 2025, calculated by StatisticsGlobal from Cloudflare data. Source: Cloudflare, From Googlebot to GPTBot: who's crawling your site in 2025 (1 July 2025, accessed 12 November 2026) https://blog.cloudflare.com/from-googlebot-to-gptbot-whos-crawling-your-site-in-2025/
- Meta-ExternalAgent took a 19% share of AI-only crawler traffic in May 2025, according to Cloudflare. Source: Cloudflare, From Googlebot to GPTBot: who's crawling your site in 2025 (1 July 2025, accessed 12 November 2026) https://blog.cloudflare.com/from-googlebot-to-gptbot-whos-crawling-your-site-in-2025/
- ByteDance's Bytespider had a 7.2% share of AI-only crawler traffic in May 2025, down from 42% in May 2024, according to Cloudflare. Source: Cloudflare, From Googlebot to GPTBot: who's crawling your site in 2025 (1 July 2025, accessed 12 November 2026) https://blog.cloudflare.com/from-googlebot-to-gptbot-whos-crawling-your-site-in-2025/
- Bytespider's share of AI-only crawler traffic fell 34.8 percentage points between May 2024 and May 2025, calculated by StatisticsGlobal from Cloudflare data. Source: Cloudflare, From Googlebot to GPTBot: who's crawling your site in 2025 (1 July 2025, accessed 12 November 2026) https://blog.cloudflare.com/from-googlebot-to-gptbot-whos-crawling-your-site-in-2025/
- GPTBot had a 28.1% share of AI-only bot traffic in July 2025, up from 11.9% in July 2024, according to Cloudflare. Source: Cloudflare, The crawl-to-click gap: Cloudflare data on AI bots, training, and referrals (29 August 2025, accessed 12 November 2026) https://blog.cloudflare.com/crawlers-click-ai-bots-training/
- ClaudeBot had a 23.3% share of AI-only bot traffic in July 2025, up from 15% in July 2024, according to Cloudflare. Source: Cloudflare, The crawl-to-click gap: Cloudflare data on AI bots, training, and referrals (29 August 2025, accessed 12 November 2026) https://blog.cloudflare.com/crawlers-click-ai-bots-training/
- Googlebot made 50% of all AI and search crawler requests in May 2025, up from 30% in May 2024, according to Cloudflare. Source: Cloudflare, From Googlebot to GPTBot: who's crawling your site in 2025 (1 July 2025, accessed 12 November 2026) https://blog.cloudflare.com/from-googlebot-to-gptbot-whos-crawling-your-site-in-2025/
- Googlebot made 39% of all AI and search crawler traffic in July 2025, according to Cloudflare. Source: Cloudflare, The crawl-to-click gap: Cloudflare data on AI bots, training, and referrals (29 August 2025, accessed 12 November 2026) https://blog.cloudflare.com/crawlers-click-ai-bots-training/
- Googlebot generated over 28% of verified bot traffic on Cloudflare's network in 2025, according to Cloudflare. Source: Cloudflare, The 2025 Cloudflare Radar Year in Review (15 December 2025, accessed 12 November 2026) https://blog.cloudflare.com/radar-2025-year-in-review/
- GPTBot generated about 7.5% of verified bot traffic on Cloudflare's network in 2025, the second highest of any bot, according to Cloudflare. Source: Cloudflare, The 2025 Cloudflare Radar Year in Review (15 December 2025, accessed 12 November 2026) https://blog.cloudflare.com/radar-2025-year-in-review/
- PerplexityBot crawl requests rose 157,490% between May 2024 and May 2025, from a very low base, according to Cloudflare. Source: Cloudflare, From Googlebot to GPTBot: who's crawling your site in 2025 (1 July 2025, accessed 12 November 2026) https://blog.cloudflare.com/from-googlebot-to-gptbot-whos-crawling-your-site-in-2025/
- ChatGPT-User requests rose 2,825% between May 2024 and May 2025, reaching 1.3% of AI and search crawling, according to Cloudflare. Source: Cloudflare, From Googlebot to GPTBot: who's crawling your site in 2025 (1 July 2025, accessed 12 November 2026) https://blog.cloudflare.com/from-googlebot-to-gptbot-whos-crawling-your-site-in-2025/
- GoogleOther made 36% of AI crawler traffic on Fastly's network in January 2026, according to Fastly. Source: Fastly, Fastly Threat Insights Report 26Q1: AI, Bots, and the Agentic Future of the Web (April 2026, accessed 12 November 2026) https://learn.fastly.com/rs/025-XKO-469/images/Fastly-Threat-Insights-Report-26Q1.pdf
- Meta's AI crawler made 28% of AI crawler traffic on Fastly's network in January 2026, according to Fastly. Source: Fastly, Fastly Threat Insights Report 26Q1: AI, Bots, and the Agentic Future of the Web (April 2026, accessed 12 November 2026) https://learn.fastly.com/rs/025-XKO-469/images/Fastly-Threat-Insights-Report-26Q1.pdf
- Huawei's PetalBot made 18% of AI crawler traffic on Fastly's network in January 2026, according to Fastly. Source: Fastly, Fastly Threat Insights Report 26Q1: AI, Bots, and the Agentic Future of the Web (April 2026, accessed 12 November 2026) https://learn.fastly.com/rs/025-XKO-469/images/Fastly-Threat-Insights-Report-26Q1.pdf
- GoogleOther, Meta's AI crawler and PetalBot together made 82% of AI crawler traffic on Fastly's network in January 2026, calculated by StatisticsGlobal from Fastly data. Source: Fastly, Fastly Threat Insights Report 26Q1: AI, Bots, and the Agentic Future of the Web (April 2026, accessed 12 November 2026) https://learn.fastly.com/rs/025-XKO-469/images/Fastly-Threat-Insights-Report-26Q1.pdf
- GPTBot made 10.8% of crawler traffic to Europe-based Cloudflare customers in October 2025, behind Googlebot and Bingbot, according to Cloudflare. Source: Cloudflare, The 2025 Cloudflare Radar Year in Review (15 December 2025, accessed 12 November 2026) https://blog.cloudflare.com/radar-2025-year-in-review/
- GPTBot made 14.0% of crawler traffic to Asia-based Cloudflare customers in October 2025, second only to Googlebot, according to Cloudflare. Source: Cloudflare, The 2025 Cloudflare Radar Year in Review (15 December 2025, accessed 12 November 2026) https://blog.cloudflare.com/radar-2025-year-in-review/
- Googlebot made 45.5% of crawler traffic to North America-based Cloudflare customers in October 2025, according to Cloudflare. Source: Cloudflare, The 2025 Cloudflare Radar Year in Review (15 December 2025, accessed 12 November 2026) https://blog.cloudflare.com/radar-2025-year-in-review/
- Retail and computer software sites drew just over 40% of AI crawler traffic on Cloudflare's network in October 2025, according to Cloudflare. Source: Cloudflare, The 2025 Cloudflare Radar Year in Review (15 December 2025, accessed 12 November 2026) https://blog.cloudflare.com/radar-2025-year-in-review/

Table: Share of AI-only crawler traffic on Cloudflare's network by bot, May 2024 and May 2025

| Bot | Operator | May 2024 | May 2025 |
|---|---|---|---|
| GPTBot | OpenAI | 5% | 30% |
| ClaudeBot | Anthropic | 27% | 21% |
| Meta-ExternalAgent | Meta | not in top 5 | 19% |
| Amazonbot | Amazon | 21% | 11% |
| Bytespider | ByteDance | 42% | 7.2% |
| Applebot | Apple | 4.1% | not in top 5 |

Source: Cloudflare, "From Googlebot to GPTBot: who's crawling your site in 2025". Top 5 bots in each month only.

## AI crawler crawl-to-referral ratios

The crawl-to-referral ratio counts how many pages a platform crawls for each visitor it sends back. A high ratio means a site gives content away and gets little traffic in return.

Anthropic crawled about 38,066 pages for every visitor it referred in July 2025. That was down 86.7% from January, after Claude added web search with links, but it was still 34.9 times OpenAI's ratio. OpenAI in turn crawled 202 times as many pages per referral as Google. Perplexity moved the wrong way, from 54.6 to 194.8.

TollBit measures something similar from publishers' own logs. European publishers on its network got one human referral for every 179 AI bot scrapes in H1 2026, over 3 times worse than North American sites. The ratio worsened 51% from Q1 to Q2 2026. TollBit only counts bots that identify themselves, so the real number of scrapes is likely higher.

The two measures are not the same. Cloudflare compares HTML requests from a platform's crawlers with visits whose referrer is that platform, across its whole network. TollBit counts scrapes and referrals on publisher sites that use its service. Both point the same way: AI platforms crawl far more than they send back, while Google still sent about 88% of referral traffic in 2026, according to Cloudflare.

- Anthropic crawled about 38,066 pages for every visitor it referred in July 2025, down from 286,930 in January 2025, according to Cloudflare. Source: Cloudflare, The crawl-to-click gap: Cloudflare data on AI bots, training, and referrals (29 August 2025, accessed 12 November 2026) https://blog.cloudflare.com/crawlers-click-ai-bots-training/
- OpenAI crawled about 1,091 pages for every visitor it referred in July 2025, according to Cloudflare. Source: Cloudflare, The crawl-to-click gap: Cloudflare data on AI bots, training, and referrals (29 August 2025, accessed 12 November 2026) https://blog.cloudflare.com/crawlers-click-ai-bots-training/
- Perplexity crawled about 195 pages for every visitor it referred in July 2025, up from 54.6 in January 2025, according to Cloudflare. Source: Cloudflare, The crawl-to-click gap: Cloudflare data on AI bots, training, and referrals (29 August 2025, accessed 12 November 2026) https://blog.cloudflare.com/crawlers-click-ai-bots-training/
- Microsoft crawled 40.7 pages for every visitor it referred in July 2025, according to Cloudflare. Source: Cloudflare, The crawl-to-click gap: Cloudflare data on AI bots, training, and referrals (29 August 2025, accessed 12 November 2026) https://blog.cloudflare.com/crawlers-click-ai-bots-training/
- Google crawled 5.4 pages for every visitor it referred in July 2025, according to Cloudflare. Source: Cloudflare, The crawl-to-click gap: Cloudflare data on AI bots, training, and referrals (29 August 2025, accessed 12 November 2026) https://blog.cloudflare.com/crawlers-click-ai-bots-training/
- Anthropic's crawl-to-referral ratio was 34.9 times OpenAI's in July 2025, calculated by StatisticsGlobal from Cloudflare data. Source: Cloudflare, The crawl-to-click gap: Cloudflare data on AI bots, training, and referrals (29 August 2025, accessed 12 November 2026) https://blog.cloudflare.com/crawlers-click-ai-bots-training/
- OpenAI's crawl-to-referral ratio was 5.6 times Perplexity's in July 2025, calculated by StatisticsGlobal from Cloudflare data. Source: Cloudflare, The crawl-to-click gap: Cloudflare data on AI bots, training, and referrals (29 August 2025, accessed 12 November 2026) https://blog.cloudflare.com/crawlers-click-ai-bots-training/
- OpenAI crawled 202 times as many pages per referral as Google in July 2025, calculated by StatisticsGlobal from Cloudflare data. Source: Cloudflare, The crawl-to-click gap: Cloudflare data on AI bots, training, and referrals (29 August 2025, accessed 12 November 2026) https://blog.cloudflare.com/crawlers-click-ai-bots-training/
- European publishers received one human referral for every 179 AI bot scrapes in H1 2026, over 3 times worse than North American sites, according to TollBit. Source: TollBit, State of the Bots 2026 Q1 & Q2: The Bad Bots (2026, covers January to June 2026, accessed 12 November 2026) https://tollbit.com/bots/
- The AI scrape-to-referral ratio on European publisher sites was 150 to 1 in Q1 2026, according to TollBit. Source: TollBit, State of the Bots 2026 Q1 & Q2: The Bad Bots (2026, covers January to June 2026, accessed 12 November 2026) https://tollbit.com/bots/
- The AI scrape-to-referral ratio on European publisher sites was 227 to 1 in Q2 2026, according to TollBit. Source: TollBit, State of the Bots 2026 Q1 & Q2: The Bad Bots (2026, covers January to June 2026, accessed 12 November 2026) https://tollbit.com/bots/
- The AI scrape-to-referral ratio on European publisher sites worsened 51% from Q1 to Q2 2026, calculated by StatisticsGlobal from TollBit data. Source: TollBit, State of the Bots 2026 Q1 & Q2: The Bad Bots (2026, covers January to June 2026, accessed 12 November 2026) https://tollbit.com/bots/
- European publisher sites saw about 1 AI bot scrape for every 33 human visits in H1 2026, according to TollBit. Source: TollBit, State of the Bots 2026 Q1 & Q2: The Bad Bots (2026, covers January to June 2026, accessed 12 November 2026) https://tollbit.com/bots/
- Google accounted for about 88% of referral traffic to websites in 2026, according to Cloudflare. Source: Cloudflare, Content Independence Day, one year on: building the business model for the agentic Internet (1 July 2026, accessed 12 November 2026) https://blog.cloudflare.com/agentic-internet-bot-report/

Table: Pages crawled per referred visitor by platform, Cloudflare network, January and July 2025

| Platform | January 2025 | July 2025 | Change |
|---|---|---|---|
| Anthropic | 286,930 | 38,066 | -86.7% |
| OpenAI | 1,217 | 1,091 | -10.4% |
| Perplexity | 54.6 | 194.8 | +256.7% |
| Microsoft | 38.5 | 40.7 | +5.7% |
| Yandex | 15.5 | 21.4 | +38.3% |
| Google | 3.8 | 5.4 | +43% |
| ByteDance | 18.0 | 0.9 | -95% |

Source: Cloudflare, "The crawl-to-click gap", crawl-to-refer ratio table. Ratios rounded to whole numbers above 1,000.

## AI crawler traffic by purpose

Cloudflare sorts AI crawling into training, search and user actions. Training has grown from most of AI crawling to about half of all crawling.

Training made 79% of AI crawling on Cloudflare in July 2025, up 7 points in a year. Search fell from 26% to 17%. By June 2026, Cloudflare said 52% of all crawler requests were for AI training, up from 22% in spring 2025. That later figure uses all crawlers as the base, including search engines, so it is not directly comparable with the 79%.

User action crawling is small but growing fastest. It happens when a chatbot fetches a page to answer a live question. Cloudflare says it grew more than 15 times over in 2025. On Fastly, OpenAI's ChatGPT-User and SearchBot made 97% of this fetcher traffic in January 2026, and 57% of fetcher requests went for content that was not cached.

Over 36% of crawler activity in 2026 came from mixed-use crawlers that blend search, agent use and training, according to Cloudflare. That matters for site owners. A mixed-use bot cannot be allowed for search and blocked for training. For more on OpenAI's products, see our [ChatGPT statistics](https://statisticsglobal.org/chatgpt-statistics/).

- 52% of crawler requests on Cloudflare's network were for AI training in June 2026, up from 22% in spring 2025, according to Cloudflare. Source: Cloudflare, Content Independence Day, one year on: building the business model for the agentic Internet (1 July 2026, accessed 12 November 2026) https://blog.cloudflare.com/agentic-internet-bot-report/
- The AI training share of crawler requests rose 30 percentage points between spring 2025 and June 2026, calculated by StatisticsGlobal from Cloudflare data. Source: Cloudflare, Content Independence Day, one year on: building the business model for the agentic Internet (1 July 2026, accessed 12 November 2026) https://blog.cloudflare.com/agentic-internet-bot-report/
- Mixed-use crawlers that blend search, agent use and training made over 36% of crawler activity in 2026, according to Cloudflare. Source: Cloudflare, Content Independence Day, one year on: building the business model for the agentic Internet (1 July 2026, accessed 12 November 2026) https://blog.cloudflare.com/agentic-internet-bot-report/
- Training made up 79% of AI crawling in July 2025, up from 72% in July 2024, according to Cloudflare. Source: Cloudflare, The crawl-to-click gap: Cloudflare data on AI bots, training, and referrals (29 August 2025, accessed 12 November 2026) https://blog.cloudflare.com/crawlers-click-ai-bots-training/
- Training's share of AI crawling rose 7 percentage points between July 2024 and July 2025, calculated by StatisticsGlobal from Cloudflare data. Source: Cloudflare, The crawl-to-click gap: Cloudflare data on AI bots, training, and referrals (29 August 2025, accessed 12 November 2026) https://blog.cloudflare.com/crawlers-click-ai-bots-training/
- Search made up 17% of AI crawling in July 2025, down from 26% in July 2024, according to Cloudflare. Source: Cloudflare, The crawl-to-click gap: Cloudflare data on AI bots, training, and referrals (29 August 2025, accessed 12 November 2026) https://blog.cloudflare.com/crawlers-click-ai-bots-training/
- User actions made up 3.2% of AI crawling in July 2025, up from 2% in July 2024, according to Cloudflare. Source: Cloudflare, The crawl-to-click gap: Cloudflare data on AI bots, training, and referrals (29 August 2025, accessed 12 November 2026) https://blog.cloudflare.com/crawlers-click-ai-bots-training/
- The user action share of AI crawling grew 60% between July 2024 and July 2025, from 2% to 3.2%, calculated by StatisticsGlobal from Cloudflare data. Source: Cloudflare, The crawl-to-click gap: Cloudflare data on AI bots, training, and referrals (29 August 2025, accessed 12 November 2026) https://blog.cloudflare.com/crawlers-click-ai-bots-training/
- AI user action crawling grew more than 15 times over during 2025, according to Cloudflare. Source: Cloudflare, The 2025 Cloudflare Radar Year in Review (15 December 2025, accessed 12 November 2026) https://blog.cloudflare.com/radar-2025-year-in-review/
- OpenAI's ChatGPT-User and SearchBot made 97% of AI fetcher traffic on Fastly's network in January 2026, according to Fastly. Source: Fastly, Fastly Threat Insights Report 26Q1: AI, Bots, and the Agentic Future of the Web (April 2026, accessed 12 November 2026) https://learn.fastly.com/rs/025-XKO-469/images/Fastly-Threat-Insights-Report-26Q1.pdf
- 57% of AI fetcher requests on Fastly's network went for content that was not cached in January 2026, according to Fastly. Source: Fastly, Fastly Threat Insights Report 26Q1: AI, Bots, and the Agentic Future of the Web (April 2026, accessed 12 November 2026) https://learn.fastly.com/rs/025-XKO-469/images/Fastly-Threat-Insights-Report-26Q1.pdf

Table: Share of AI crawling by purpose on Cloudflare's network, July 2024 and July 2025

| Purpose | July 2024 | July 2025 |
|---|---|---|
| Training | 72% | 79% |
| Search | 26% | 17% |
| User action | 2% | 3.2% |

Source: Cloudflare, "The crawl-to-click gap". Shares of AI crawling only.

## Websites blocking AI crawlers

robots.txt is the main way sites ask AI crawlers to stay away. Use is growing but still a minority practice among large sites, and compliance is voluntary.

About 14% of top 10,000 domains with a robots.txt file had rules aimed at AI bots in June 2025, according to Cloudflare. GPTBot was the most disallowed bot, by 312 domains, or 8.2% of the sample. It was also the most explicitly allowed, by 61 domains, so disallows outnumbered allows about 5 to 1.

The sources that matter most to AI training are the most likely to opt out. The Data Provenance Initiative found that over 28% of the most actively maintained sources in the C4 training dataset were fully restricted by robots.txt by 2024, against over 5% of all its tokens.

robots.txt is a request, not a lock. TollBit found that robots.txt rules on the median European site were ignored 2.8 times as often as on North American sites in H1 2026. In a separate test, 19 of the top 20 UK sites could be scraped by at least one of 14 commercial scraping tools. Cloudflare counted more than 50 publisher-AI content agreements by July 2026, a sign that some sites now license rather than block.

- About 14% of top 10,000 domains with a robots.txt file had rules targeting AI bots in June 2025, 546 of 3,816 domains, according to Cloudflare. Source: Cloudflare, From Googlebot to GPTBot: who's crawling your site in 2025 (1 July 2025, accessed 12 November 2026) https://blog.cloudflare.com/from-googlebot-to-gptbot-whos-crawling-your-site-in-2025/
- GPTBot was disallowed by 312 top 10,000 domains in June 2025, 250 fully and 62 partially, the most of any bot, according to Cloudflare. Source: Cloudflare, From Googlebot to GPTBot: who's crawling your site in 2025 (1 July 2025, accessed 12 November 2026) https://blog.cloudflare.com/from-googlebot-to-gptbot-whos-crawling-your-site-in-2025/
- 8.2% of top 10,000 domains with a robots.txt file disallowed GPTBot in June 2025, calculated by StatisticsGlobal from Cloudflare data. Source: Cloudflare, From Googlebot to GPTBot: who's crawling your site in 2025 (1 July 2025, accessed 12 November 2026) https://blog.cloudflare.com/from-googlebot-to-gptbot-whos-crawling-your-site-in-2025/
- GPTBot was explicitly allowed by 61 top 10,000 domains in June 2025, 18 fully and 43 partially, according to Cloudflare. Source: Cloudflare, From Googlebot to GPTBot: who's crawling your site in 2025 (1 July 2025, accessed 12 November 2026) https://blog.cloudflare.com/from-googlebot-to-gptbot-whos-crawling-your-site-in-2025/
- Top 10,000 domains disallowed GPTBot 5.1 times as often as they explicitly allowed it in June 2025, calculated by StatisticsGlobal from Cloudflare data. Source: Cloudflare, From Googlebot to GPTBot: who's crawling your site in 2025 (1 July 2025, accessed 12 November 2026) https://blog.cloudflare.com/from-googlebot-to-gptbot-whos-crawling-your-site-in-2025/
- Over 5% of all tokens in the C4 AI training dataset were fully restricted by robots.txt by 2024, according to the Data Provenance Initiative. Source: Data Provenance Initiative, Consent in Crisis: The Rapid Decline of the AI Data Commons (arXiv 2407.14933) (July 2024, accessed 12 November 2026) https://arxiv.org/abs/2407.14933
- Over 28% of the most actively maintained, critical sources in C4 were fully restricted by robots.txt by 2024, according to the Data Provenance Initiative. Source: Data Provenance Initiative, Consent in Crisis: The Rapid Decline of the AI Data Commons (arXiv 2407.14933) (July 2024, accessed 12 November 2026) https://arxiv.org/abs/2407.14933
- 45% of the C4 AI training dataset was restricted by terms of service crawling rules by 2024, according to the Data Provenance Initiative. Source: Data Provenance Initiative, Consent in Crisis: The Rapid Decline of the AI Data Commons (arXiv 2407.14933) (July 2024, accessed 12 November 2026) https://arxiv.org/abs/2407.14933
- The median European site's robots.txt rules were ignored 2.8 times more often than North American sites' in H1 2026, according to TollBit. Source: TollBit, State of the Bots 2026 Q1 & Q2: The Bad Bots (2026, covers January to June 2026, accessed 12 November 2026) https://tollbit.com/bots/
- Common Crawl scraped about 74% of sites on the TollBit network in H1 2026, roughly double its reach from Q1 to Q2, according to TollBit. Source: TollBit, State of the Bots 2026 Q1 & Q2: The Bad Bots (2026, covers January to June 2026, accessed 12 November 2026) https://tollbit.com/bots/
- 19 of the top 20 UK sites ranked by Press Gazette could be scraped by at least one of 14 tested scraping tools in 2026, according to TollBit. Source: TollBit, State of the Bots 2026 Q1 & Q2: The Bad Bots (2026, covers January to June 2026, accessed 12 November 2026) https://tollbit.com/bots/
- Median AI scrapes per site were 4 times higher on European sites than North American sites in H1 2026, according to TollBit. Source: TollBit, State of the Bots 2026 Q1 & Q2: The Bad Bots (2026, covers January to June 2026, accessed 12 November 2026) https://tollbit.com/bots/
- More than 50 publisher-AI content agreements had been signed between 2023 and July 2026, according to Cloudflare. Source: Cloudflare, Content Independence Day, one year on: building the business model for the agentic Internet (1 July 2026, accessed 12 November 2026) https://blog.cloudflare.com/agentic-internet-bot-report/

Table: How widely AI crawlers are restricted, by source, scope and year

| Source | Scope | Year | Finding |
|---|---|---|---|
| Cloudflare | 3,816 top 10,000 domains with robots.txt | 2025 | 14% had rules aimed at AI bots; 312 disallowed GPTBot |
| StatisticsGlobal calculation | Same sample | 2025 | 8.2% disallowed GPTBot; 5.1 disallows per explicit allow |
| Data Provenance Initiative | About 14,000 domains behind AI training sets | 2024 | Over 28% of critical C4 sources fully restricted by robots.txt |
| Reuters Institute | Top 15 news sites in 10 countries | 2023 | 48% blocked OpenAI, 24% blocked Google AI |
| TollBit | Publisher sites on TollBit | 2026 | European robots.txt rules ignored 2.8 times as often as North American |

Sources: Cloudflare, Data Provenance Initiative, Reuters Institute, TollBit. GPTBot shares calculated by StatisticsGlobal.

## News publishers blocking AI bots

News publishers moved first. The best independent country-level data is the Reuters Institute's 2023 study of top news sites.

48% of top news websites in 10 countries blocked OpenAI's crawlers by the end of 2023, and 24% blocked Google's AI crawler. So OpenAI was twice as likely to be blocked. Almost every site that blocked Google AI, 97%, also blocked OpenAI. No site in the study unblocked once it had blocked.

Country differences were large. 79% of top US news sites blocked OpenAI, against 20% in Mexico and Poland, a 59 point gap. Germany led for Google, at 60%. Legacy print outlets were the most likely to block OpenAI, at 57%, 26 points more than digital-born outlets.

These figures are from 2023, before many licensing deals were signed. They are the latest independent cross-country numbers we could check. Newer network data from Cloudflare and TollBit covers all kinds of sites, not just news.

- 48% of top news websites in 10 countries blocked OpenAI's crawlers by the end of 2023, according to the Reuters Institute. Source: Reuters Institute, How many news websites block AI crawlers? (Reuters Institute for the Study of Journalism, University of Oxford) (22 February 2024, accessed 12 November 2026) https://reutersinstitute.politics.ox.ac.uk/how-many-news-websites-block-ai-crawlers
- 24% of top news websites in 10 countries blocked Google's AI crawler by the end of 2023, according to the Reuters Institute. Source: Reuters Institute, How many news websites block AI crawlers? (Reuters Institute for the Study of Journalism, University of Oxford) (22 February 2024, accessed 12 November 2026) https://reutersinstitute.politics.ox.ac.uk/how-many-news-websites-block-ai-crawlers
- Top news websites were twice as likely to block OpenAI's crawlers as Google's AI crawler in 2023, calculated by StatisticsGlobal from Reuters Institute data. Source: Reuters Institute, How many news websites block AI crawlers? (Reuters Institute for the Study of Journalism, University of Oxford) (22 February 2024, accessed 12 November 2026) https://reutersinstitute.politics.ox.ac.uk/how-many-news-websites-block-ai-crawlers
- 79% of top US news websites blocked OpenAI's crawlers by the end of 2023, the highest of 10 countries, according to the Reuters Institute. Source: Reuters Institute, How many news websites block AI crawlers? (Reuters Institute for the Study of Journalism, University of Oxford) (22 February 2024, accessed 12 November 2026) https://reutersinstitute.politics.ox.ac.uk/how-many-news-websites-block-ai-crawlers
- 20% of top news websites in Mexico and in Poland blocked OpenAI's crawlers by the end of 2023, the lowest of 10 countries, according to the Reuters Institute. Source: Reuters Institute, How many news websites block AI crawlers? (Reuters Institute for the Study of Journalism, University of Oxford) (22 February 2024, accessed 12 November 2026) https://reutersinstitute.politics.ox.ac.uk/how-many-news-websites-block-ai-crawlers
- The share of top news sites blocking OpenAI was 59 percentage points higher in the US than in Mexico or Poland in 2023, calculated by StatisticsGlobal from Reuters Institute data. Source: Reuters Institute, How many news websites block AI crawlers? (Reuters Institute for the Study of Journalism, University of Oxford) (22 February 2024, accessed 12 November 2026) https://reutersinstitute.politics.ox.ac.uk/how-many-news-websites-block-ai-crawlers
- 60% of top German news websites blocked Google's AI crawler by the end of 2023, the highest of 10 countries, according to the Reuters Institute. Source: Reuters Institute, How many news websites block AI crawlers? (Reuters Institute for the Study of Journalism, University of Oxford) (22 February 2024, accessed 12 November 2026) https://reutersinstitute.politics.ox.ac.uk/how-many-news-websites-block-ai-crawlers
- 57% of legacy print news websites blocked OpenAI's crawlers by the end of 2023, against 31% of digital-born outlets, according to the Reuters Institute. Source: Reuters Institute, How many news websites block AI crawlers? (Reuters Institute for the Study of Journalism, University of Oxford) (22 February 2024, accessed 12 November 2026) https://reutersinstitute.politics.ox.ac.uk/how-many-news-websites-block-ai-crawlers
- Legacy print news sites were 26 percentage points more likely than digital-born outlets to block OpenAI's crawlers in 2023, calculated by StatisticsGlobal from Reuters Institute data. Source: Reuters Institute, How many news websites block AI crawlers? (Reuters Institute for the Study of Journalism, University of Oxford) (22 February 2024, accessed 12 November 2026) https://reutersinstitute.politics.ox.ac.uk/how-many-news-websites-block-ai-crawlers
- 97% of top news websites that blocked Google's AI crawler in 2023 also blocked OpenAI's crawlers, according to the Reuters Institute. Source: Reuters Institute, How many news websites block AI crawlers? (Reuters Institute for the Study of Journalism, University of Oxford) (22 February 2024, accessed 12 November 2026) https://reutersinstitute.politics.ox.ac.uk/how-many-news-websites-block-ai-crawlers

Table: Share of top news websites blocking AI crawlers by type of outlet, 10 countries, end of 2023

| Outlet type | Blocking OpenAI | Blocking Google AI |
|---|---|---|
| Legacy print | 57% | 32% |
| TV and radio broadcasters | 48% | 19% |
| Digital-born | 31% | 17% |
| All top news sites | 48% | 24% |

Source: Reuters Institute, University of Oxford, "How many news websites block AI crawlers?" (2024).

## How these sources measure AI crawlers

Cloudflare data comes from traffic across its network, which it says carries more than 20% of the web. It identifies crawlers by user agent and by its verified bot list, and assigns each a purpose from operator disclosures. Cloudflare sells AI crawler controls, so it has a commercial interest in the topic. Its posts use different bases: AI-only crawling, AI plus search crawling, HTML requests and verified bot traffic. We keep each figure on its own base.

Fastly data covers its Next-Gen WAF and Bot Management customers in January 2026 only. Fastly does not count search engine crawlers as AI bots. TollBit data comes from publisher sites on its network and counts only bots that identify themselves. Both companies sell bot management or content licensing.

The Reuters Institute read archived robots.txt files for the 15 top news sites in each of 10 countries through 2023. The Data Provenance Initiative audited about 14,000 domains behind AI training datasets for 2023 to 2024. These two are independent research, but older.

Cloudflare's 2026 report, Fastly's report and TollBit's page had no Wayback Machine snapshot when we checked. We saved a copy of each quoted figure, with a file fingerprint, on this site. Cloudflare Radar's live charts could not be read as text, so this page uses only figures stated in Cloudflare's published posts.


## Frequently asked questions

### How much web traffic comes from AI crawlers?

AI bots other than Googlebot made an average of 4.2% of HTML requests on Cloudflare's network in 2025, according to Cloudflare. Googlebot, which crawls for both search and AI, made another 4.5%.

### Which AI crawler is the most active?

GPTBot had a 30% share of AI-only crawler traffic on Cloudflare's network in May 2025, up from 5% in May 2024, according to Cloudflare. On Fastly in January 2026, GoogleOther led with 36% of AI crawler traffic.

### How many pages do AI crawlers take for each visitor they send?

Anthropic crawled about 38,066 pages for every visitor it referred in July 2025, OpenAI about 1,091 and Google 5.4, according to Cloudflare. TollBit found European publishers got one human referral per 179 AI scrapes in H1 2026.

### Is AI crawling for training going up or down?

52% of crawler requests on Cloudflare's network were for AI training in June 2026, up from 22% in spring 2025, according to Cloudflare. Among AI crawlers alone, training rose from 72% to 79% between July 2024 and July 2025.

### How many websites block AI crawlers?

About 14% of top 10,000 domains with a robots.txt file had rules targeting AI bots in June 2025, according to Cloudflare. Among news sites, 48% of top sites in 10 countries blocked OpenAI by the end of 2023, according to the Reuters Institute.

### Which countries' news sites block AI crawlers most?

79% of top US news websites blocked OpenAI's crawlers by the end of 2023, the highest of 10 countries, according to the Reuters Institute. Mexico and Poland were lowest at 20%. Germany led for Google's AI crawler at 60%.

## Sources

1. Cloudflare, Content Independence Day, one year on: building the business model for the agentic Internet, 1 July 2026, accessed 12 November 2026. https://blog.cloudflare.com/agentic-internet-bot-report/
2. Cloudflare, The crawl-to-click gap: Cloudflare data on AI bots, training, and referrals, 29 August 2025, accessed 12 November 2026. https://blog.cloudflare.com/crawlers-click-ai-bots-training/
3. TollBit, State of the Bots 2026 Q1 & Q2: The Bad Bots, 2026, covers January to June 2026, accessed 12 November 2026. https://tollbit.com/bots/
4. Cloudflare, The 2025 Cloudflare Radar Year in Review, 15 December 2025, accessed 12 November 2026. https://blog.cloudflare.com/radar-2025-year-in-review/
5. Reuters Institute, How many news websites block AI crawlers? (Reuters Institute for the Study of Journalism, University of Oxford), 22 February 2024, accessed 12 November 2026. https://reutersinstitute.politics.ox.ac.uk/how-many-news-websites-block-ai-crawlers
6. Cloudflare, From Googlebot to GPTBot: who's crawling your site in 2025, 1 July 2025, accessed 12 November 2026. https://blog.cloudflare.com/from-googlebot-to-gptbot-whos-crawling-your-site-in-2025/
7. Fastly, Fastly Threat Insights Report 26Q1: AI, Bots, and the Agentic Future of the Web, April 2026, accessed 12 November 2026. https://learn.fastly.com/rs/025-XKO-469/images/Fastly-Threat-Insights-Report-26Q1.pdf
8. Data Provenance Initiative, Consent in Crisis: The Rapid Decline of the AI Data Commons (arXiv 2407.14933), July 2024, accessed 12 November 2026. https://arxiv.org/abs/2407.14933
