The blocked customer files no report

While researching this post, our automated fetches of three security-industry websites were refused with 403s and bot challenges. Researching the cost of over-blocking was, itself, obstructed by over-blocking. That is the whole subject in miniature: the industry that sells blocking cannot see how much it blocks, because the thing it blocks — a visitor who was actually fine — leaves no trace.

Fraud that gets through your defences generates artifacts. A chargeback arrives with a case number. A spam run fills a moderation queue. A brute-force attempt lands in a log. Every false *negative* produces a measurable, attributable, alertable event, and someone gets asked about it.

A customer wrongly turned away produces nothing. They see a 403 page or an unsolvable challenge, and they do what you would do: they close the tab and buy somewhere else. No ticket, no alert, no dashboard number. The false-positive side of the ledger is invisible by construction — and a system where one error type screams and the other is silent will always drift toward more blocking. This post is about that drift: what it costs, why it happens, and what our own production data says about how much of your traffic is actually dangerous.

TL;DR

• Over-blocking is invisible by design. Fraud that gets through creates a chargeback and an alert; a real customer wrongly blocked creates silence. Nobody tunes against a number they can't see.
• Very little traffic is actually risky. Across 93,414 unique IPs scored by our production API, 75.4% carry no risk signal at all and only 2.2% score at or above the default block threshold.
• The datacenter trap is the big one: 30.3% of those IPs carry a datacenter flag, and 27.9% are datacenter with *no* VPN, proxy, or Tor signal. Block "datacenter" outright and you turn away roughly 28% of traffic to act on a risky pool of about 2%.
• Status codes decide your SEO fate. Google removes content that returns 403s; a worldwide outage serving 5xx barely registers. If your firewall says "no" with a 403, it is quietly deindexing you.
• VPN users are ordinary customers. Around a third of US adults use a VPN; a quarter of users cite secure online *shopping* as a reason.
• We shipped every one of these mistakes ourselves — a preset that blocked a customer's entire VPN audience, 403s served to Googlebot, a "Block VPN" toggle that did nothing, a monitor mode that blocked. The fix each time was structural, not a threshold tweak.

What over-blocking actually costs

Revenue, silently. A blocked checkout is a sale your analytics never records as lost — the visitor bounces before the funnel starts, so they don't even show up as an abandoned cart. The only symptom is a conversion rate slightly lower than it should be, indistinguishable from a hundred other causes. The e-commerce industry talks a great deal about false declines at the payment step; the same failure exists one layer earlier, at the door, and it's measured even less. We won't attach a number to it — the widely circulated multiples don't trace to primary sources — but the mechanism is not in dispute: every wrongly blocked visitor is revenue that exits without a receipt.

Search visibility, catastrophically. Google's guidance here is unambiguous, and we'll spend a full section on it below: serve enough 403s to Googlebot and your pages leave the index.

Accessibility and abandonment. The friction you add instead of blocking has its own bill. In a USENIX Security 2023 study by Searles et al. (1,400 participants, 14,000 CAPTCHAs), between 18% and 45% of participants abandoned the study after being shown their first CAPTCHA — and the bots the CAPTCHAs exist to stop outperformed the humans on both solving time and accuracy across the types tested. The WebAIM Million survey (February 2026, one million home pages) found reCAPTCHA present on 9.6% of pages, and those pages averaged 7.7 *more* accessibility errors than the norm. Challenges are not free; they are a tax paid disproportionately by real people.

Support load and trust. The small fraction of blocked customers who do get in touch arrive angry, and your support team can rarely tell them *why* they were blocked — especially if the block was logged under a reason that doesn't name the real cause. More on that failure below, because we shipped it.

Our own numbers: how much traffic is actually risky?

The strongest argument against blanket blocking is simply the shape of real traffic. Here is the distribution across the IPs our production API has scored — measured 13 August 2026 from our own ip_history table, aggregate only, no customer identified:

• 93,414 unique IP addresses observed
• 70,414 (75.4%) score 0 — no risk signal of any kind
• 2,100 (2.2%) score at or above the default block threshold in our WordPress and WooCommerce plugins
• 16 IPs in the entire dataset land in the 15-point band just above that threshold

That last number deserves a moment. The risk distribution is not a smooth curve you slice with a carefully chosen threshold — it is a barbell. Traffic clusters hard at "no signal" and hard at "multiple strong signals," with almost nothing in between. Move your threshold ten points and you change the outcome for a handful of IPs out of ninety-three thousand. The endless tuning debate about *where exactly* to set the score threshold is mostly theater; what actually changes outcomes is which categories of signal you act on. (For how the score itself is built, see IP risk scoring explained.)

And that is where the trouble starts.

The datacenter trap

From the same dataset:

• 28,315 IPs (30.3%) carry a datacenter flag
• 26,054 (27.9%) are datacenter with no VPN, proxy, or Tor signal whatsoever

A datacenter flag means one thing: this IP is allocated to hosting infrastructure rather than a consumer ISP. That is a fact about *where the packets come from*, not a verdict about *who sent them*. The datacenter category contains, among much else:

• corporate egress — plenty of businesses route office traffic through cloud infrastructure
• every commercial VPN exit (which is a privacy choice, not an attack)
• uptime monitors, link-preview fetchers, and accessibility tooling
• security scanners your own vendors run on your behalf
• every major search engine crawler

It also contains genuinely hostile automation — scrapers, credential-stuffing bots, carding scripts. The problem is the ratio. A site that hard-blocks "datacenter IPs" turns away roughly 28% of its traffic in order to act on a genuinely risky pool of about 2%. That is not a security posture; it is a fourteen-to-one bet against your own visitors. The residential-vs-datacenter distinction matters enormously for *weighting* a decision — a checkout from a hosting range deserves more scrutiny than one from a residential ISP — but as a standalone block rule it is the single most expensive mistake in this category.

We know, because we made a version of it. For a period, our own strictest plugin preset served 403s to Googlebot and Bingbot. The cause was ordering, not intent: the datacenter rule was evaluated before the crawler check, and search engines crawl from datacenter IPs, so the crawler exemption was never reached. The API classified the crawlers correctly the whole time — verified, risk zero — and the plugin blocked them anyway. Any site running that preset was slowly removing itself from search. We shipped a verified-crawler exemption that runs before every policy rule; the lesson generalizes to any stack where "datacenter" can outrank "known good."

The status code you say "no" with decides your SEO

If your blocking layer touches crawlers at all — and if it blocks datacenter IPs, it does — then the single most consequential configuration detail is the HTTP status code it answers with.

Google's guidance, from Gary Illyes on the Search Central blog (February 2023, "Don't use 403s or 404s for rate limiting"), could not be plainer: "All 4xx HTTP status codes (again, except 429) will cause your content to be removed from Google Search." A 403 does not tell Google "come back later." It tells Google "this content is forbidden," and Google eventually believes you. Illyes has also said that firewalls and CDNs blocking Googlebot is "by far the most common issue in my inbox," and that "in the vast majority of the cases the blockage is unintended" — and, of automatically generated firewall rules: "Sometimes they perceive crawler traffic as malicious and block it."

Now set that against what happened on 18 November 2025, when a global Cloudflare outage served 5xx errors from a large fraction of the web simultaneously. John Mueller's assessment: with 5xx, "Google crawling slows down, but it'll ramp back up… even then, those will pop back in fairly quickly." A worldwide outage was a search non-event. A firewall 403 removes you from the index. The damage is not downtime — it is which status code you choose to say "no" with.

The cost of getting this wrong is measurable — when anyone bothers to measure it. In August 2024, Barry Schwartz of Search Engine Roundtable documented an incident on his own site: AWS S3 began serving 404s to Googlebot for his images while they loaded fine in browsers. The result on his own analytics: an 83% drop in image impressions and a 76% drop in image-search clicks; two months after the fix, still down 16% and 26%. One publisher, measuring one site — and, in his words: "The weird part is that I saw zero public complaints about the issue." Silent failure, again, all the way down.

Two practical notes complete this section. First, if you exempt verified crawlers — and you should — verify properly (reverse-DNS or Google's published IP ranges, not the spoofable user-agent string; our bot detection guide covers how). Second, even doing it properly by IP list breaks if the list goes stale: Google updated its published crawler ranges on 4 February 2025, and WAFs pinned to old snapshots began rate-limiting the new ranges. Mueller's advice: "We push the IP json files automatically… If you need to alert internally on those files, feel free to poll them." Re-poll, or your correctness has an expiry date. The same applies to AI crawlers, where the range lists are newer and change more often.

One more measurement gap worth naming: Ahrefs' Patrick Stox surveyed ~140 million sites in May 2025 and found the most-blocked crawlers in robots.txt were MJ12bot (6.49%), SemrushBot (6.34%), and AhrefsBot (6.31%). The interesting part is his own caveat — the study "doesn't include any other block types such as firewalls or IP blocks." The industry's largest crawl-blocking survey is blind to exactly the layer this post is about. Nobody is measuring firewall-level over-blocking at scale, because it doesn't announce itself.

VPN users are customers, not suspects

The Security.org 2025 VPN Consumer Report (1,009 US adults, fielded June 2025) found 32% of US adults currently use a VPN, rising to around 40% among 18–29-year-olds. Their stated reasons: general privacy (60%), general security (57%), a job that requires it (25%) — and secure online shopping (25%). A quarter of VPN users are on a VPN *specifically because they're buying things*. Block VPNs at checkout and you are selecting against your most security-conscious customers at the exact moment they're paying you.

The report also discloses its own limitation, which is worth repeating because it makes the point better than the headline number: their survey platform blocks VPN users, so the figure likely *under*-reports actual usage. Even the people measuring VPN adoption can't see all of it — because their own tooling over-blocks.

In our data, 2,267 IPs (2.4%) carry a VPN flag. A VPN flag deserves weight in a score, and a "block VPN" toggle is a legitimate policy choice for some sites (here's how to make that choice well on WordPress). What it should never be is an accident — which brings us to our own history.

We shipped all of these bugs. Here's why the category drifts this way

This is not a competitor hit piece. Over-blocking is the *default failure mode* of fraud and bot tooling — including ours — and the reason is the asymmetry from the top of this post: false negatives generate alerts, false positives generate silence, so every incentive in development, testing, and support tuning pushes one direction. Four exhibits from our own engineering history:

The churned customer. A customer running a media-download site with a VPN-heavy audience installed our plugin on its strictest preset and churned within days. Investigation showed every VPN visitor was being blocked — not by the VPN rule, but *via the generic risk-score threshold*, and logged with the reason risk_threshold, which never mentioned VPN. The site owner could see blocks happening but had no way to diagnose what was causing them. We later cut the VPN weight so a VPN flag alone no longer crosses the default block threshold; blocking VPNs is now something an admin chooses explicitly, not a side effect of a score.

The crawler 403s. Covered above: our strict preset blocked Googlebot and Bingbot because the datacenter rule ran before the crawler exemption.

The toggle that did nothing. We shipped a "Block VPN" toggle that had no effect, because the datacenter rule matched first and returned before the VPN branch was ever reached — nearly every VPN exit is also a datacenter IP. The toggle was real; the code path was unreachable. Worse, turning the toggle *off* also did nothing, and the logs attributed every block to "datacenter," so the admin believed they were blocking bots while blocking their audience.

Monitor mode that blocked. We shipped a "monitor only" preset that left one blocking rule active, because the preset omitted a key and applying it only wrote the keys present. "Watch, don't touch" quietly kept touching.

Notice the pattern: none of these were bad thresholds. All four were *structural* — rule ordering, unreachable branches, mislabeled log reasons, incomplete presets. And all four were invisible for the same reason: the people being wrongly blocked didn't file bug reports. The one that surfaced fastest did so because a paying customer disappeared.

A practical program for measuring and reducing false positives

1. Monitor before you block. Every category rule should have a monitor mode, and you should run it for at least a week of real traffic before enforcing. If a rule would have flagged 28% of your visitors, you want to learn that from a log, not from your conversion rate.
2. Act on categories, not just the score. The barbell distribution means threshold-tuning is low-leverage. Decide per category: hard-block what real customers never use (open proxies at checkout, Tor at checkout, high-velocity repeat offenders), score or challenge what they do (VPN, bare datacenter). Country rules deserve the same scrutiny — blunt geographic blocks share most of the failure modes described here.
3. Exempt verified crawlers from every policy rule — verified by reverse-DNS or published IP ranges, never by user-agent — and keep the exemption *ahead of* the datacenter and geography rules in evaluation order. Ordering bugs are how we served Googlebot 403s.
4. Say "no" with the right status code. If you rate-limit or defer crawlers, use 429 or 503 — never 403 or 404. A 4xx is an eviction notice, not a speed bump.
5. Prefer holding and flagging over hard blocking where the workflow allows it. An order held for review costs you minutes; a blocked checkout costs you the customer. Our chargeback playbook covers where review queues beat blocks.
6. Log the actual reason. "Blocked: risk_threshold" told our customer nothing. Logs should name the rule that fired *and* the signals behind it, so a support ticket — on the rare occasion one arrives — can be answered in one look, and so you can audit what your own rules are doing.
7. Re-poll your allowlists. Crawler IP ranges change; Google pushes updates and expects you to poll. A correct exemption with a stale list becomes an incorrect block on a schedule you don't control.
8. Count the invisible column. You can't measure blocked customers directly, but you can proxy it: track your block rate by category over time, alert when it jumps, and reconcile blocked-traffic volume against what you know about your audience. If your "datacenter" blocks are 30% of traffic and your fraud losses are unchanged, the rule isn't protecting you — it's just blocking.

The honest close

Every failure described in this post — the invisible churn, the crawler 403s, the dead toggle, the monitor mode that blocked — is one we shipped, found, and fixed in our own product. We're writing this not because we're above the failure mode but because we've measured how easy it is to fall into: the category's error signals are asymmetric, and unless you deliberately engineer visibility into the false-positive side, you will over-block and never know.

The data says most of your traffic is fine: three-quarters of the IPs we score carry no risk signal at all, and the genuinely dangerous pool is about 2%. Good IP intelligence exists to let you act on that 2% precisely — not to hand you a bigger hammer for the other 98%.

If you want the signals to do that precisely — per-category flags, verified-crawler identification, and a risk score you can monitor before you enforce — the WordPress and WooCommerce plugins let you set every category independently, and their Recommended preset monitors VPN traffic rather than blocking it, for all the reasons above. The free API tier is enough to audit what your current rules are really blocking.