The Lead

Between Bytespider, Amazonbot, ClaudeBot, and GPTBot, the four busiest AI crawlers collectively accessed roughly 40% of the top one million internet properties by June 2024, yet only 2.98% of those sites were actively blocking AI bots at the time — per Cloudflare's analysis when it shipped its one-click "Block AI bots" toggle on July 3, 2024. That gap — two-out-of-three sites getting scraped by an AI crawler, fewer than one-in-thirty sites doing anything about it — is the subject of this post. It is also the context for *The New York Times* v. *Microsoft* and *OpenAI*, filed December 27, 2023 and still in active litigation as of this writing. Whether you consider AI crawlers a licensing opportunity, a scraping nuisance, or a copyright-infringement-in-progress, your first problem is the same: who is actually crawling your site, and what can you do about it?

TL;DR

• The big AI crawlers are GPTBot, ChatGPT-User, ClaudeBot, Google-Extended, PerplexityBot, Bytespider, Amazonbot, and CCBot. Each operator publishes a user-agent token, most publish IP ranges, and RFC 9309 formalises the robots.txt standard they are *supposed* to obey.
• Robots.txt is advisory, not binding. RFC 9309 is explicit: "These rules are not a form of access authorization." Perplexity has been caught bypassing it repeatedly; Bytespider has a long history of non-compliance.
• Verification is the same pattern Google uses for Googlebot: forward-confirmed reverse DNS (FCrDNS) against the crawler's canonical hostname, plus published IP range cross-checks.
• Your three strategies are block, monetise, or throttle. News publishers and content-licensees block. Larger publishers are increasingly monetising via TollBit or direct licensing. Smaller sites can throttle and log.
• Layered defence wins. Robots.txt for the cooperative, user-agent filters at the web server for the lazy, IP-range enforcement for the actively hostile. Real detection happens at the IP layer.

Who's Actually Crawling Your Site in 2026

There are dozens of AI crawlers, but a handful account for the overwhelming majority of AI-scraping traffic. If your robots.txt and WAF rules cover the eight below, you've handled roughly 95% of the volume.

GPTBot harvests pages for OpenAI's training corpus. ChatGPT-User is distinct — it fires when a ChatGPT user prompts the model to browse the live web for an answer, meaning even if you've blocked GPTBot, ChatGPT-User may still fetch your content in real time. OAI-SearchBot crawls for ChatGPT Search results. Blocking all three requires three separate robots.txt stanzas.

ClaudeBot gathers data for Anthropic's Claude models. The Anthropic help article explicitly states the robots.txt directive and maintains a JSON file of current crawler IPs at claude.com/crawling/bots.json.

Google-Extended is not a crawler. It is a robots.txt control token that Google introduced to let publishers opt out of having their content used for Gemini and Vertex AI training, without affecting Googlebot's indexing for search. Blocking it does not affect your search rankings. Crawling itself still happens from Googlebot's IPs — Google-Extended is purely a gate on downstream use of the already-fetched content. The original announcement is on Google's blog.

PerplexityBot and Perplexity-User feed Perplexity's answer engine. As discussed below, Perplexity has had well-documented compliance problems.

Bytespider is ByteDance's (TikTok's parent) crawler. It has historically ignored robots.txt and has no published IP ranges, which makes it the hardest of the major AI crawlers to enforce against cleanly.

Amazonbot powers Alexa and Amazon's AI training. CCBot is Common Crawl's open-data crawler — Common Crawl is a non-profit whose data is frequently used as a training-set seed by many AI labs, so blocking CCBot is a one-step way to refuse a broad category of downstream training uses.

A maintained superset of these lives at Known Agents (formerly Dark Visitors), which publishes machine-readable lists of AI user-agent strings and their operators — useful if you want to stay current without writing the monitoring yourself.

Why robots.txt Is a Polite Request, Not a Block

The Robots Exclusion Protocol dates back to 1994 and was formalised as RFC 9309 in September 2022. RFC 9309's text is blunt on its own enforceability: "*These rules are not a form of access authorization.*" The specification uses advisory language throughout — crawlers "are requested to honor" the rules, which is not the same thing as "must."

In practice, this means robots.txt is load-bearing for two groups of crawlers. The first: operators who view compliance as reputational capital — Google, Microsoft, OpenAI, Anthropic (for ClaudeBot), and most of the mainstream AI labs publicly commit to honoring it and take violations seriously when raised. The second: polite academic and non-profit crawlers like CCBot, which have no financial incentive to cheat.

Everyone else? The picture is messier. Perplexity has been the 2024-era lightning rod. Robb Knight's June 2024 investigation demonstrated that Perplexity summarised his content even after he had blocked PerplexityBot via both robots.txt and server-side rules. A follow-up Wired investigation found Perplexity's IPs accessed Condé Nast properties hundreds of times in three months despite explicit blocks, apparently using undisclosed IP ranges and spoofed user-agent strings to bypass both robots.txt and IP blocks. Perplexity's CEO responded by suggesting the company "relied on third-party web crawlers" for the contested traffic — a response widely read as deflection rather than explanation.

Bytespider has a similar but older reputation. ByteDance's crawler has been observed ignoring robots.txt for years across multiple community audits, and Bytespider does not publish IP ranges, which makes enforcement harder.

The conclusion: treat robots.txt as a hygiene baseline. It filters cooperative crawlers quickly and cheaply. It filters no one else. If your policy matters, you need IP and user-agent enforcement at a layer below robots.txt — at the CDN, at the origin web server, or in your application.

Verifying Real Bots from Spoofed UAs

A spoofed user-agent string is a free attack. Sending User-Agent: GPTBot costs the attacker nothing and can be used to probe for crawler-specific treatment (cache warming, preferential rate limits, different rendering). Conversely, an attacker impersonating Googlebot to bypass rate limits is a well-known pattern.

Both categories are defeated by the same technique Google has used for Googlebot verification for over a decade: forward-confirmed reverse DNS (FCrDNS). The algorithm:

1. Reverse-DNS the incoming IP. You get a hostname (the PTR record).
2. Check that the hostname ends in the crawler's canonical suffix — for GPTBot this is OpenAI's published range list rather than PTR, so GPTBot validation is IP-list-based; for ClaudeBot you'd check against Anthropic's bot list at claude.com/crawling/bots.json; for Googlebot, the hostname ends in .googlebot.com or .google.com.
3. Forward-DNS the hostname back to an IP. It must match the incoming IP.

If any step fails, the request is spoofed. Real crawlers pass all three. Here's what the validation looks like against a Googlebot IP at the shell:

# Step 1: reverse DNS the visitor IP
$ dig +short -x 66.249.64.1
crawl-66-249-64-1.googlebot.com.

# Step 2: confirm the hostname ends in a known crawler suffix
$ echo "crawl-66-249-64-1.googlebot.com" | grep -E "\.(googlebot|google)\.com\.?\$"
crawl-66-249-64-1.googlebot.com

# Step 3: forward DNS the hostname back
$ dig +short crawl-66-249-64-1.googlebot.com
66.249.64.1

# Match — verified Googlebot.

For AI crawlers specifically, most operators publish machine-readable IP range files that you can fetch and diff nightly:

# Pull current OpenAI / Anthropic / Common Crawl IP lists
curl -s https://openai.com/gptbot.json | jq '.prefixes[].ipv4Prefix'
curl -s https://openai.com/chatgpt-user.json | jq '.prefixes[].ipv4Prefix'
curl -s https://claude.com/crawling/bots.json | jq '.ipv4_prefixes[].ipv4_prefix'
curl -s https://index.commoncrawl.org/ccbot.json | jq '.prefixes[].ipv4Prefix'

Pipe those into your WAF's allowlist and you have IP-level verification that is independent of whatever string the visitor put in User-Agent. This is the same mechanic Googlebot uses — the only difference is that for AI crawlers, most labs publish the IP list as JSON rather than relying on PTR-based verification. Both methods work; the IP-list method is simpler to automate.

Block, Monetise, or Throttle? Three Strategies

Once you know who's crawling, the policy question is what to do about it. There are three clean strategies and a lot of muddled middle ground.

Full Block

The strategy for operators who see AI scraping as straightforwardly harmful. News publishers with fresh-content value (The New York Times, Condé Nast, Reuters, Axel Springer) have largely chosen this route, often while simultaneously pursuing legal remedy. Full block means: all major AI crawler user-agent tokens disallowed in robots.txt, IP-level enforcement at the WAF for the published ranges, and user-agent regex rules in the web server for spoofers. Recovery is easy — you just lift the rules — so there's little downside to trying it for a quarter and measuring the impact on referral traffic (for most sites, near-zero, because AI crawlers' citation-back-to-source behaviour is already weak and uneven).

Monetise

The strategy for publishers whose content has clear economic value and who want to capture that value rather than forfeit it. TollBit, ScalePost, and Cloudflare's "Pay per Crawl" beta offer metered access: AI crawlers pay a per-request fee in exchange for structured, authorised access to content. The economics are still nascent — per-fetch rates are small fractions of a cent — but aggregate revenue at scale is material for large sites, and more importantly, the licensing relationship creates a clean legal basis for the crawling that sidesteps the copyright-infringement questions of the block-and-sue approach. Direct licensing deals (the OpenAI × Financial Times agreement, OpenAI × Axel Springer, OpenAI × News Corp, OpenAI × Reddit) are the larger-deal version of the same idea.

Throttle and Log

The strategy for smaller sites. You don't have the legal budget to sue, and your content isn't individually licensable at meaningful rates, so you do what's cheap: robots.txt saying no, then rate-limit anything claiming to be an AI crawler to something like 1 request per 10 seconds. You log the violations in case the legal landscape changes. You don't spend engineering time on enforcement beyond that. For a blog, a SaaS marketing site, or a small community forum, this is often the right economics.

A Practical Implementation: Layered Defence

Here's what a reasonable end-to-end setup looks like. Three layers, each a fallback for the one above.

Layer 1: robots.txt

User-agent: GPTBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: OAI-SearchBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: Amazonbot
Disallow: /

User-agent: CCBot
Disallow: /

Place this at https://example.com/robots.txt. Handles cooperative crawlers.

Layer 2: Web-Server User-Agent Rules

For the crawlers that ignore robots.txt but still honestly identify themselves, filter at the web server. Nginx:

map \$http_user_agent \$ai_crawler {
    default 0;
    "~*GPTBot" 1;
    "~*ChatGPT-User" 1;
    "~*ClaudeBot" 1;
    "~*PerplexityBot" 1;
    "~*Perplexity-User" 1;
    "~*Bytespider" 1;
    "~*CCBot" 1;
}

server {
    if (\$ai_crawler) {
        return 403;
    }
}

Caddy is simpler:

@aicrawlers header_regexp User-Agent (?i)(GPTBot|ClaudeBot|PerplexityBot|Bytespider|CCBot)
respond @aicrawlers 403

This stops operators who send the real user-agent but do not honor robots.txt. It does not stop spoofers.

Layer 3: IP-Range Enforcement for Bypassers

For Perplexity-style actors who spoof the user-agent or rotate through undisclosed IPs, you need IP-level intelligence. The minimum useful thing is to call an IP-reputation service on incoming requests and branch on whether the IP is a known crawler (verified or unverified), a hosting-ASN (a likely source of scraping infrastructure), or flagged for other abuse signals.

import requests, os

def should_allow(ip: str, user_agent: str) -> bool:
    r = requests.post(
        "https://predax.io/api/v1/check/ip",
        headers={"X-API-Key": os.environ["PREDAX_API_KEY"]},
        json={"ip": ip},
        timeout=2,
    ).json()

    classification = r.get("classification", {})
    crawler = r.get("crawler", {})

    # Block known-bad high-risk IPs regardless of UA
    if classification.get("risk_score", 0) >= 75:
        return False

    # If the UA claims to be a verified bot but the IP says it isn't, block it
    claims_bot = any(tok in user_agent for tok in ("GPTBot", "ClaudeBot", "Googlebot"))
    if claims_bot and not crawler.get("verified", False):
        return False

    # Block hosting-ASN traffic claiming to be a browser (common scraper pattern)
    if classification.get("is_hosting") and "Mozilla" in user_agent and not classification.get("is_crawler"):
        return False

    return True

The exact field names vary by provider; the pattern is universal. The point is to cross-check the user-agent's *claims* against the IP's *reality*, and only let requests through when the two agree.

A deeper treatment of general bot detection — of which AI scraping is one category — lives in our 2026 bot traffic detection guide.

What About Fair Use and the Legal Fight?

Two cases matter for context, and a pile of others are following in their wake.

*The New York Times* v. *Microsoft* and *OpenAI* was filed in the Southern District of New York on December 27, 2023. The complaint alleges that OpenAI unlawfully used millions of *Times* articles to train its models, and that ChatGPT will reproduce substantial portions of *Times* content verbatim when prompted. The lawsuit seeks "billions of dollars in statutory and actual damages" and the destruction of ChatGPT's training dataset that contains *Times* material. In April 2025 Judge Sidney Stein denied OpenAI's motion to dismiss large parts of the complaint, allowing the case to proceed on the merits — including claims related to contributory infringement by Microsoft as the infrastructure provider.

*Getty Images* v. *Stability AI* addresses the image side. Getty alleges Stability AI trained Stable Diffusion on Getty's licensed library without authorisation, with parallel cases in the US (District of Delaware) and UK (English High Court). The UK trial concluded in mid-2025 with a split verdict that narrowed but did not eliminate Getty's theory; appeals are ongoing.

The unresolved question in both is fair use — whether training a generative model on copyrighted works transforms them sufficiently to qualify as fair use, or whether it is straightforward reproduction at a scale that existing doctrine was not designed to handle. There is no settled answer yet, and the outcome will shape what "blocking AI crawlers" means legally as well as technically.

This is not legal advice. If your content strategy turns on how these cases resolve, you need a lawyer who practices IP in your jurisdiction.

How Predax Helps

AI scraping looks like a content problem at the top of the stack, but at the network layer it is the same problem as any other automated traffic: is this IP who it claims to be, and do I recognise the network it's coming from? Predax answers both in a single call.

Every /api/v1/check/ip response includes `classification.is_crawler` (Boolean: did this IP resolve to a known crawler suffix?), and a top-level `crawler` object with `crawler.verified` (Boolean: did forward-confirmed reverse DNS succeed?) and `crawler.name` (the matched crawler, when verified — googlebot, bingbot, yandexbot, baiduspider). Combined with `network.asn`, `network.as_name`, and the `classification.is_hosting` flag, the response lets you write exactly the kind of cross-check policy described in Layer 3 above: if the user-agent claims Googlebot but crawler.verified is false, the request is spoofed; if the IP belongs to a hosting ASN and claims a browser user-agent, it's probably a scraper.

For publishers and store operators on WordPress, the WordPress plugin's verified-crawler detection wires the same checks into your site without code. The plugin recognises legitimate Googlebot / Bingbot / Yandex / Baidu traffic via FCrDNS and can be configured to short-circuit rate limits and WAF rules for verified crawlers while blocking spoofers that claim the same user-agent from unverified IPs. Paired with robots.txt and user-agent filtering, that's a three-layer defence against both polite and impolite AI crawlers, without having to maintain the IP lists yourself.

Your admin dashboard shows every crawler verification event — what UA claimed what, which IP it came from, whether verification succeeded — so you can audit what's actually hitting your site rather than guessing from access logs. View pricing for plan tiers, including a generous free tier that's sufficient for most small sites running the AI-crawler defence described in this post.

Frequently Asked Questions

Is it legal to block AI crawlers?

Yes, in virtually every jurisdiction. You own the server, and serving or refusing a request is a business decision you get to make — subject only to the usual limits (anti-discrimination law, sector-specific regulation, and whatever contracts you've signed with upstream providers). Robots Exclusion Protocol as formalised by RFC 9309 is advisory for the crawler, but you are under no obligation to let unauthenticated automated clients in. The harder questions are not about blocking being permissible — they are about enforcement (how do you prove an IP was a bot?) and about contracts (did you promise public access in a user-facing SLA or ToS?). Neither is a legal obstacle to having the policy.

Does blocking GPTBot stop ChatGPT from citing my site?

No — and this is the single most common misunderstanding about OpenAI's user-agent taxonomy. GPTBot is the training crawler; it fetches pages so OpenAI can use them as training data. ChatGPT-User is a different user-agent, fired when a ChatGPT user prompts the model to browse the live web to answer a question. Blocking GPTBot stops future training-data capture but does not stop ChatGPT-User from retrieving your content in real time to answer a user query, because the two crawlers honor different robots.txt directives and use different IP ranges. To stop both, you need Disallow: / lines for each user-agent token, and ideally IP-level enforcement as a belt-and-braces.

What's the difference between GPTBot and ChatGPT-User?

GPTBot is OpenAI's batch crawler used to gather training data for future model versions. It fetches large volumes of pages on a schedule and honors the GPTBot robots.txt token. ChatGPT-User is a live, user-triggered fetcher — it only fires when a ChatGPT user specifically asks the model to browse the web to answer their query. It honors the ChatGPT-User robots.txt token and uses a distinct set of IP egress ranges. Treating them as one bot leads to the common failure mode of blocking GPTBot, patting yourself on the back, and then discovering ChatGPT is still quoting your articles back to users. Block both tokens independently, and consider blocking both sets of IP ranges at your WAF if you want the policy to be enforced, not just requested.

Does Cloudflare block AI bots by default?

Not by default — but Cloudflare shipped a one-click toggle in July 2024 that makes it trivial to enable. Per Cloudflare's announcement, the "AI Scrapers and Crawlers" toggle lives in the Security > Bots section of the dashboard and is available to customers on every plan including the free tier. It blocks a curated list of AI crawler fingerprints that Cloudflare maintains, which they have stated will be "automatically be updated over time as we see new fingerprints of offending bots." If you already use Cloudflare this is the fastest single-click action you can take. Note that the toggle is off by default; flipping it is your decision.

Related guides

• Bot traffic in 2026: detection guide — the wider bot-detection picture this fits inside.
• ASN-based blocking explained — most AI scrapers operate from a small number of hosting ASNs.
• Form spam in 2026 — what scrapers turn into once they start submitting forms.
• How IP risk scoring works — turning AI-bot signals into a single score you can act on.

See pricing to add AI-bot-aware filtering across your site.