This report uses the Crawl-to-Refer ratio—the number of crawled pages per referral—to examine changes from July 2024 to the end of 2025. Data is drawn from the Cloudflare August 2025 study and the SearchSignal February 2026 benchmark report; Tow Center entries are excluded due to duplicate links.
Research Background
AI crawlers may serve model training, search indexing, or user-request invocations—none of which inherently equates to users clicking on source links.
This ratio is calculated by dividing the number of crawled pages by the number of referrals. An increase indicates more machine reads per return visit, but it does not explain citation or conversion outcomes.
Training Crawls Account for a Growing Share, Widening the Gap Between Bot and User Traffic
According to the Cloudflare August 2025 study, the share of training crawls rose from 72% in July 2024 to 79% in July 2025, averaging 82% over the prior six months; during the same period, search-related crawling fell from 26% to 17%, while user-action-triggered crawling increased from 2% to 3.2%.
A considerable portion of new crawl activity serves training purposes rather than sending users back to websites. Looking only at bot request volumes makes it easy to confuse “being visited by machines” with “being discovered by users.”
Crawl-to-Refer Ratios Show AI Platforms Read Far More Than They Refer
In Cloudflare's fixed sample, Anthropic logged 38,065.7 crawls per referral in July 2025; Perplexity rose from 54.6 in January to 194.8 in July, an increase of 256.7%. Over the same period, OpenAI recorded 1,091.4 and Google 5.4.
The SearchSignal 2026 benchmark report summarizes these figures by product name as Google 5.4, ChatGPT 1,091, and Claude 38,066. This portion originates from the same Cloudflare sample period; both naming conventions are retained in the table to avoid double-counting or ranking discrepancies.
| Channel or Platform | Reporting Period | Crawls per Referral | Data Source | Methodology Note |
|---|---|---|---|---|
| July 2025 | 5.4 | SearchSignal | Aggregated from Cloudflare | |
| ChatGPT | July 2025 | ~1,091 | SearchSignal | Corresponds to OpenAI methodology |
| Claude | July 2025 | ~38,066 | SearchSignal | Corresponds to Anthropic methodology |
| Anthropic | July 2025 | 38,065.7 | Cloudflare | Not double-counted with Claude |
| Perplexity | July 2025 | 194.8 | Cloudflare | +256.7% growth from Jan–Jul |
AI answers can directly satisfy some user needs, meaning users may not open the source page. AI crawl volume reflects how often content is accessed by machines, while referral traffic reflects whether users return to the website—these are not the same growth curve.
AI Referral Traffic Is Growing Rapidly, but Its Share Remains Limited
SearchSignal, aggregating 2024–2025 research from Similarweb, BrightEdge, and Digiday, notes a 357% year-over-year increase in AI referral traffic; however, samples from Conductor, SE Ranking, and Ahrefs show it accounts for roughly 0.1%–1.08% of total website traffic. Given differences in samples and attribution rules, these ranges should not be merged into a single market average.
The Conductor sample covers 13,770 domains and 3.3 billion sessions; from May to September 2025, AI referrals accounted for 2.8% of traffic to IT and technology websites. High growth does not mean AI has become a primary traffic source across all industries.
Trend Observation: Crawling, Citation, and Visits Need to Be Measured in Layers
Cloudflare's cross-regional news sample shows referral traffic from Google was 9% lower in March 2025 and 15% lower in April 2025 compared to January. The timing roughly coincides with the Gemini 2.0 update and AI Overviews expansion, but causality cannot be inferred from this correlation.
Crawling, citation, brand mentions, referral visits, and conversions represent different stages and do not naturally grow at the same rate.
Research Implications
Businesses can log crawler agents, purposes, and page types; monitor citations or brand mentions within ChatGPT, Gemini, Google AI Mode, Perplexity, and Google AI Overview; and separately tag identifiable AI referral visits.
Combining GEO audits, Schema checks, performance monitoring, and trend analysis makes it possible to observe the path from “crawled → understood → cited → visited.”
When AI crawling, brand mentions, and referral traffic fall out of sync, audit your data methodology stage by stage. aipogeo lets you check your real visibility across 6 major AI engines in 1 minute.
Related Questions
Why doesn't AI crawler traffic to website content generate more referral traffic?
Some crawling serves training or indexing purposes and is not triggered by user requests; AI answers may also directly fulfill user needs.
What is the Crawl-to-Refer ratio?
It represents the number of crawled pages per website referral and does not directly indicate citation quality or conversions.
How do the Crawl-to-Refer ratios of ChatGPT, Claude, and Perplexity differ?
In Cloudflare's July 2025 sample, the figures were 1,091.4, 38,065.7, and 194.8, respectively—these should be interpreted within the same timeframe.
Which metrics should websites track after being crawled frequently by AI bots?
Focus on crawl purpose, citations, brand mentions, referral visits, and conversions to pinpoint changes at each layer.