This report uses the Crawl-to-Refer ratio—the number of crawled pages per referral—to track changes from July 2024 to late 2025. Data is drawn from Cloudflare's August 2025 study and SearchSignal's February 2026 benchmark report; the Tow Center entry is excluded due to duplicate links.
Research Background
AI crawlers may serve model training, search indexing, or user-request calls—none of which equate to users clicking source links.
This ratio is calculated by dividing the number of crawled pages by referral counts. A rising ratio indicates more machine reads per return visit, but it does not indicate citation or conversion outcomes.
Training Crawl Share Rises, Further Diverging Bot Traffic from User Visits
According to Cloudflare's August 2025 study, the share of training crawls rose from 72% in July 2024 to 79% in July 2025, with a six-month average of 82% prior; over the same period, search-related usage dropped from 26% to 17%, and user-action-triggered crawls rose from 2% to 3.2%.
A significant portion of new crawl activity serves training rather than bringing users back to websites. Looking at bot request volume alone makes it easy to conflate “being visited by machines” with “being discovered by users.”
Crawl-to-Refer Ratios Show AI Platforms Read Far More Than They Refer
Cloudflare's fixed sample shows that in July 2025, Anthropic logged 38,065.7 crawls per referral; Perplexity rose from 54.6 in January to 194.8 in July, a 256.7% increase. During the same period, OpenAI logged 1,091.4 crawls per referral, and Google logged 5.4.
SearchSignal's 2026 benchmark report summarizes these figures by product name: Google 5.4, ChatGPT 1,091, Claude 38,066. This section is derived from Cloudflare's same-period sample; both naming conventions are preserved in the table to avoid double-counting or ranking.
| Channel or Platform | Reporting Period | Crawled Pages per Referral | Data Source | Methodology Notes |
|---|---|---|---|---|
| July 2025 | 5.4 | SearchSignal | Aggregates Cloudflare data | |
| ChatGPT | July 2025 | ~1,091 | SearchSignal | Corresponds to OpenAI metrics |
| Claude | July 2025 | ~38,066 | SearchSignal | Corresponds to Anthropic metrics |
| Anthropic | July 2025 | 38,065.7 | Cloudflare | Not double-counted with Claude |
| Perplexity | July 2025 | 194.8 | Cloudflare | 256.7% increase from Jan–Jul |
AI answers can directly satisfy some user queries, so users may not open source pages. AI crawl volume reflects how often content is accessed by machines, while referral traffic reflects whether users return to your site—these are not on the same growth curve.
AI Referral Traffic Grows Quickly but Still Accounts for a Limited Share
After aggregating 2024–2025 research from Similarweb, BrightEdge, and Digiday, SearchSignal notes that AI referral traffic grew 357% year-over-year; however, different samples from Conductor, SE Ranking, and Ahrefs show it accounts for only 0.1%–1.08% of total site traffic. Given variations in sample sets and attribution rules, this range should not be consolidated into a single market average.
Conductor's sample covers 13,770 domains and 3.3 billion sessions; from May to September 2025, AI referrals accounted for 2.8% of IT and technology site traffic. High growth does not mean AI has become a primary traffic source across industries.
Trend Insights: Crawl, Citation, and Visits Should Be Measured in Layers
Cloudflare's cross-regional news sample shows that referral traffic from Google in March 2025 was 9% lower than January, and 15% lower in April. This timing roughly coincides with Gemini 2.0 updates and AI Overviews expansion, but causality cannot be inferred from this correlation.
Crawling, citations, brand mentions, referral visits, and conversions represent different stages and do not naturally grow proportionally.
Research Implications
Enterprises can log crawler agents, purposes, and page types; monitor citations or brand mentions in ChatGPT, Gemini, Google AI Mode, Perplexity, and Google AI Overview; and separately flag identifiable AI referral visits.
Combining GEO detection, Schema checks, performance monitoring, and trend analysis can reveal the path from “crawled—understood—cited—visited.”
When AI crawling, brand mentions, and referrals fall out of sync, check data methodologies stage by stage. aipogeo lets you view your real visibility across 6 major AI engines in just 1 minute.
Related Questions
Why doesn't AI crawler traffic to website content bring more visitors?
Some crawls serve training or indexing and are not triggered by user requests; AI answers may also directly satisfy user needs.
What is the Crawl-to-Refer ratio?
It represents the number of crawled pages per website referral and does not directly indicate citation quality or conversions.
What are the differences in Crawl-to-Refer ratios among ChatGPT, Claude, and Perplexity?
In Cloudflare's July 2025 sample, the figures were 1,091.4, 38,065.7, and 194.8 respectively—these must be interpreted within the same time frame.
Which metrics should websites track after being frequently crawled by AI bots?
Focus on crawl purposes, citations, brand mentions, referral visits, and conversions to pinpoint changes at each layer.