The Web Scraping Insider #8

Hey Web Scraping Insiders - Ian here.

Let's get to it!

⚖️🕵️‍♂️ When "Ethical" Proxies Aren't Ethical

"Ethical" has become the proxy industry's favourite word over the last few years. Every residential pool is now "ethically sourced," opt-in, consent-based, GDPR compliant. 

In most cases it's an empty term, because almost no provider will actually show you the apps their IPs come from. No public partner list, no audit trail, no independent check. It's not verification. It's marketing. And right now nobody is policing it.

The clearest sign? In just the last couple of weeks, researchers keep catching "consent-based" residential proxies hidden inside software people never knew they were running:

• 🖥️ Half the smart-TV app store is a proxy network. Spur Intelligence scanned 6,038 LG webOS and Samsung Tizen apps and found proxy SDKs in 2,058 of them, 42.5% of LG apps and 26.9% on Samsung. (Help Net Security)

• 📺 "Consent" buried in a TV remote. Bright Data's SDK quietly turns always-on smart TVs into exit nodes that relay scraping traffic for the AI industry, behind a consent dialog hidden in arrow-key menu navigation most people never see. (The Hacker News)

• 📦 Proxies preloaded at the factory. SuperBox streaming boxes sold at major US retailers ship with dormant software (Popanet) that routes strangers' traffic, including stolen credentials and account-takeover material, through home connections with no consent at all. (Plume Security Labs)

• 🚨 The FBI is now warning consumers their everyday devices are being silently turned into proxy nodes. (FBI/IC3)

None of those people meaningfully consented to anything. Yet those same IPs feed pools sold as "ethical."

👀 Our take: "Ethical" should be a claim you have to back, not a word you slap on a landing page. If a provider wants to compete on ethics, that has to mean a published partner list, a real audit trail, and independent verification: who consented, in which app, and when. 

Our bet is that over the next year or two the market goes exactly there. Anyone can print "ethically sourced." Soon they'll have to prove it, and the providers who can will pull away from the ones who just say it.

🔮 Proxy Tester: Now Benchmarks Residential Proxies Too (Free)

We just expanded the ScrapeOps Proxy Tester to cover residential proxies. It already benchmarks ~15 proxy-API providers against your exact target. Now it does the same for residential pools, so you can compare both side-by-side.

How it works: point it at your URL, and it runs real requests through each provider, tests every config they expose, then ranks them by success rate and cost per successful request, not marketing claims.

Residential is where the marketing fluff runs deepest. "30M+ IPs," "99% success rates," "best in class", none of it tells you the one number that matters: your cost per successful payload on your target. And from what we've seen across billions of requests, the CPM you pay rarely correlates with the performance you get.

Bottom line: If you're picking a residential provider without benchmarking it against the rest on your actual target, you're guessing, and probably overpaying.

👉 Try the Proxy Tester here.

🥊 The Browser Wars Are Back: People Are Rewriting Chromium Itself

For a decade, browser innovation in scraping meant building automation libraries on top of Chrome, Selenium, Puppeteer, Playwright. The browser underneath was a commodity nobody touched.

That's the shift worth paying attention to: the next wave isn't another automation library. It's people rewriting the browser itself.

Two things are forcing it. 

1. Anti-bot detection now reads deep into the browser, your TLS handshake, network stack and process behaviour, not just a JavaScript flag; so patching Chrome after launch (undetected-chromedriver, playwright-stealth) no longer holds up. 

2. Chrome is heavy: when you're running thousands of browsers for scraping, or AI agents that browse for hours, a stripped-down purpose-built browser wins on cost and speed. 

A custom browser solves both at once.

A few projects leading it:

• 🛡️ CloakBrowser patches fingerprints at the Chromium source level, not via JavaScript injection. It's pitched as a drop-in Playwright/Puppeteer replacement and claims 30/30 on the public bot-detection suites.

• ⚡ Obscura ditches Chromium entirely; a headless engine built from scratch, exposing a CDP server so Playwright still talks to it like Chrome. Its claims: ~70 MB binary, ~30 MB RAM, near-instant startup vs Chrome's 200 MB+ and ~2s. (Those are the project's own numbers, not independently benchmarked.)

• 🦊 Camoufox proves it's not just a Chromium story.. a modified Firefox spoofing fingerprints at the C++ level, with the strongest headless evasion in independent tests.

👀 Insider Take

Stealth is moving below the automation layer, where anti-bot can't easily look. Most of these projects are young and several lean on self-reported benchmarks, so don't rip out your stack yet but worth keeping your eye on things.

That's it for this #8 newsletter.

None of this replaces the boring work. Benchmark on your targets. Monitor for breakage. Measure cost-per-validated-payload, not vendor adjectives. The agents aren't waiting for permission - and neither should your testing.

Ian from ScrapeOps.