The Headline Number

Imperva's 2024 Bad Bot Report measured 49.6% of all internet traffic as non-human — the highest share they have recorded since they started tracking in 2013. Of that, 32% is classified as malicious ("bad bot") and the remaining ~18% is "good bots" like search engine crawlers, SEO tools, and uptime monitors. Put differently: out of every 100 requests hitting your website, roughly 50 are machines, and roughly 32 of those want to do something you would rather they didn't.

This has tripled from the early 2010s. Three things are driving the growth:

1. Cloud infrastructure is trivially cheap. A bot operator can spin up 10,000 EC2 instances for a weekend for less than the cost of a nice dinner.
2. Residential proxy networks are industrialised. Attackers now buy IP diversity as a commodity with 72+ million unique IPs in major pools.
3. Generative AI has lowered the cost of "good enough" bots. Imperva's 2024 data shows simple bots alone jumped from 33% to 39.6% of traffic year over year, driven partly by LLM-powered scrapers trained to defeat basic CAPTCHAs and mimic human clicking patterns.

If you run any public-facing site, you are paying to serve bots. The question is how much, what they are doing, and whether you can cut them out of the hot path without breaking legitimate automation like Googlebot.

Good Bots vs Bad Bots

The bot ecosystem splits cleanly into two halves.

Good bots you want to serve:

• Search engine crawlers (Googlebot, Bingbot, DuckDuckBot, Yandex)
• Social card generators (Facebook, LinkedIn, Twitter, Slack preview bots)
• Uptime monitors (Pingdom, UptimeRobot, StatusCake)
• Link checkers (Ahrefs, Moz, Semrush — less clearly useful but generally not malicious)
• Accessibility and performance crawlers (Lighthouse, WebPageTest)
• Your own monitoring and health checks

Bad bots you want to stop or slow:

• Scrapers: content theft, price monitoring against your wishes, competitor intelligence
• Credential stuffers: testing leaked username:password pairs against your login endpoint
• Carding bots: testing stolen credit card numbers on your checkout
• Signup/registration bots: creating fake accounts at scale for abuse
• Inventory hoarders: buying limited-edition products or booking slots before real users can
• Ad fraud bots: clicking on ads to drain your budget (if you run campaigns)
• Spam comment bots: posting links and promotional junk on any form that accepts input
• Vulnerability scanners: probing for known CVEs, path traversals, SQL injection, exposed admin paths

The two halves behave similarly at the HTTP layer — both can set a browser User-Agent, both can load JavaScript, both can solve simple interactions. The distinguishing signals are almost always on the network side rather than the request side.

Why User-Agent Detection Alone Is Useless

The 2024 ATO landscape makes this explicit: 44% of account takeover attacks target APIs directly, and those attacks set whatever User-Agent strings they want. Filtering out "Mozilla/5.0" does nothing when 90% of bad bots claim to be Chrome on macOS.

A slightly better approach is to verify User-Agents against the server's real identity. Googlebot, for example, always resolves to a *.googleusercontent.com or *.google.com PTR record, and the forward DNS lookup matches the source IP. If a request claims to be Googlebot but originates from an AWS EC2 instance, it is a fraudulent user-agent. Predax performs this forward-and-reverse DNS verification for the major search engine crawlers automatically and exposes the result as is_verified_crawler. Unverified crawlers claiming to be search engines are one of the strongest signals of malicious intent you can get.

The Network-Layer Signals That Actually Work

Real bot detection looks at the shape of the network and the behaviour over time, not single requests in isolation.

1. ASN type and reputation. Almost no legitimate end-user traffic comes from commercial datacenter ASNs. When you see classification.is_datacenter: true, you are looking at either a real cloud service (Googlebot, Slackbot, etc.) or an automated client of some kind. Verify the first, challenge or block the second.

2. Community threat scoring. An IP that other sites have reported as a bot is very likely still a bot on your site. Shared threat feeds and community reputation APIs let you catch known-bad IPs before they hit your own defences. Predax rolls third-party community reputation into the classification.risk_score and surfaces matching feeds in classification.flags and classification.reasons[] — an entry with source: "community_threat" or similar means the IP has been reported across the network and the score has been raised accordingly.

3. Request velocity per source. A real browser session loads 10–50 resources per page view. A scraper often loads only the HTML body and nothing else, then moves to the next URL immediately. Session-level velocity anomalies (high request rate, narrow resource distribution, no JavaScript execution) identify bots even when individual requests look fine.

4. TLS fingerprints. The JA3/JA4 fingerprint derived from the TLS client hello is remarkably stable per client. Modern Chrome, Firefox, and Safari each have a distinctive JA3. Python's requests library, Go's default HTTP client, curl, and every major bot framework have their own signatures. Mismatches between the User-Agent string and the JA3 (a request claiming to be Chrome with Python's TLS signature) are unambiguous bot tells.

5. Behavioural patterns. Real users have short session depths, irregular timing between actions, scroll events, mouse movements, and focus/blur events. Bots can fake each of these individually, but matching the full distribution is expensive. Client-side analytics can forward these signals to your backend for correlation with the network data.

A Practical Bot Defence in Four Layers

Layer 1 — Network reputation (free to run, ~5ms latency). Run every incoming request through an IP reputation API. Allow verified search engine crawlers, block datacenter traffic that isn't on your allowlist, challenge Tor and residential proxy traffic. Predax handles this with a single /check/ip call.

Layer 2 — Rate limiting per ASN, not per IP. Per-IP rate limiting is defeated by proxy rotation. Per-ASN limits are much harder to defeat — if an attacker has 10,000 IPs all from AS14061 DigitalOcean, you can limit that ASN collectively.

Layer 3 — Behavioural challenges on sensitive endpoints. On signup, login, checkout, and any form that sends email or creates content, require a lightweight JavaScript challenge or CAPTCHA. Keep it off your browse pages and SEO-critical content — you don't want to block Googlebot, and you definitely don't want friction on your homepage.

Layer 4 — Monitoring and triage. Track the percentage of traffic flagged by each layer over time. If your blocked-bot rate climbs suddenly, you're under attack. If your challenged-bot rate climbs, it's a smaller, smarter operation. Build dashboards, set alerts, and review monthly.

What NOT to Do

• Don't block all datacenter traffic blindly. You will block your own monitoring, legitimate APIs, and some mobile carrier gateways that happen to be classified as datacenter.
• Don't rely on CAPTCHA alone. Modern solver services route through residential proxies and solve reCAPTCHA v2 at 95%+ accuracy for fractions of a cent per solve. CAPTCHA is friction for humans and a line-item cost for attackers — not a barrier.
• Don't trust a single signal. Every signal has false positives. Decision logic should combine 3–5 independent signals before taking a block action.
• Don't hide the fact that you blocked. Return a clear 403 with a brief explanation. A silent drop wastes your bandwidth and confuses legitimate users who happen to trip a rule.

Where Predax Fits

Predax is the network layer of the four-layer stack above. A single API call per request returns all the signals a bot defence policy typically needs: verified crawler status, datacenter/residential classification, VPN/proxy/Tor flags, community threat score, ASN reputation, and an aggregated risk score from 0–100. You still need the behavioural and challenge layers on top, but the expensive part — maintaining and correlating threat feeds from dozens of upstream sources — is handled.

Half of web traffic is already bots. The question for defenders is no longer whether to care about bot detection, but where to draw the line between blocking, challenging, and allowing. The data you need to draw that line well is network data — and network data scales from a single API call.

Related guides

• AI scraper bots: GPTBot, ClaudeBot, and friends — the fastest-growing slice of bot traffic in 2026.
• ASN-based blocking explained — the highest-leverage filter for hosting-ASN bot traffic.
• How IP risk scoring works — turning bot-network signals into a usable score.
• Form spam in 2026 (beyond reCAPTCHA) — what bots do once they get past your front door.

See pricing to add bot-aware filtering across your endpoints.