👋 Hey Web Scraping Insiders - Ian here.
This month we look at:
Proxy shortlist. who actually stands out after ~9 billion pages a month
Google’s
/gotoSERP URLs. providers adapted fast, but a 200 still isn’t a complete payloadThree AMAs. BrowserAct, httpcloak, and NodeMaven
A few things we found. $1/month scraping, Chrome’s new CPU signal, ShieldFont
Let’s get to it!
🕵️♂️ Inside ScrapeOps’ Proxy Provider Shortlist
We’ve just published a behind-the-scenes look at some of the strongest proxy providers in the market.
After routing around 9 billion pages per month across 60,000 domains, we’ve seen first-hand that different providers win in different situations. This isn’t a generic ranking from first to tenth. It’s a look at who stands out, what they’re best at, and where we would actually use them.
💸 Zyte API - Best for Low-Volume Scraping
Zyte’s pay-as-you-go pricing makes it a strong choice for small or intermittent workloads. Scraping 10,000 pages can cost roughly $1–2, without being locked into a $20–50 monthly plan. It becomes competitive again at very high volumes, but the middle of the volume curve is usually better served elsewhere.
📈 Scrape.do - Best Value Web Scraping API
Scrape.do consistently delivers some of the strongest cost-to-performance we see on easy and medium-difficulty domains. It becomes particularly attractive from around 250,000 pages per month into the millions, although its domain-specific credit multipliers mean you need to check the real cost for your targets.
🔀 ScrapeOps Proxy Aggregator - Best for Maximising Cost-to-Performance
The Proxy Aggregator is designed for teams scraping many domains where no single provider consistently wins. It automatically routes requests to the cheapest provider performing well and fails over when performance changes. Full disclosure: this is our product. The trade-off is an extra routing layer and some latency, but you avoid managing and continuously evaluating multiple providers yourself.
🛡️ Scrapfly - Best for Difficult Anti-Bot Protection
Scrapfly is the specialist we would reach for when dealing with the hardest targets, including serious Cloudflare, DataDome and custom anti-bot systems. It is expensive for routine scraping, but its pricing can make sense when other providers’ success rates collapse.
📦 Oxylabs Web Unblocker - Best for Protected API Endpoints and Small Responses
Oxylabs uses bandwidth-based pricing rather than charging for every successful request. That creates a major arbitrage opportunity for small responses. A 10 KB response costs approximately $56 per million requests, compared with around $1,500 through a per-request unblocker. Once responses become much larger, especially full HTML pages, that advantage can disappear.
⚡ GeoNode Unlimited Web Scraping API - Lowest Potential Cost at Full Utilisation
GeoNode charges for concurrent threads rather than individual requests. If you have predictable traffic and can keep those threads busy around the clock, the effective cost per page can become exceptionally low. If the threads sit idle, however, the economics deteriorate quickly.
🏢 Bright Data - Best for Enterprise Procurement and Compliance
Bright Data does not necessarily offer the best cost-to-performance for any individual workload. Its advantage is breadth, global coverage, compliance infrastructure and the ability to pass through complex legal and procurement processes. For large enterprises, that organisational maturity can be worth paying for.
👀 The Takeaway
Don’t rely on headline pricing or generic success-rate claims. They rarely tell you what a provider will cost on your actual targets.
The pricing model can matter more than the provider’s brand. Per-GB pricing, pay-as-you-go plans, concurrency pricing and cost-scaled credits all create different opportunities.
There are real arbitrage opportunities. The right pricing model can make one provider dramatically cheaper than another for the same workload.
Always test the shortlist against your own domains. These categories are useful starting points, but the provider that wins in production will depend on your targets, response sizes, traffic patterns and required success rate.
🔗 Google Changed Search Result URLs. Web Scraping Providers Adapted Fast.
One of the biggest web scraping stories this month was Google replacing destination URLs in its search results with google.com/goto?url=... redirects.
The Web Scraping Club wrote a great analysis of the initial change, then followed up with a more detailed update in its latest Crawl of Fame report.
The key findings:
🧩 Google does not appear to serve the same markup to every client. Reports vary depending on account status, browser and other classifier signals.
🔁 The links are easy to resolve, but there are traps. A plain GET request returns the final URL, while HEAD returns a misleading 200 without a destination. Tokens also change on every impression, so there is no reusable lookup table.
⚡ The resolution process is fast. The Web Scraping Club resolved around 110 links per second at concurrency 20, with no rate limits.
🛠️ The major providers adapted quickly. DataForSEO says it resolves 99.99% of organic SERPs, while SerpApi restored direct URLs within days.
💸 The practical impact has been minimal so far. There is an additional request and some engineering work, but the cost appears small enough for providers to absorb without a meaningful impact on customers.
👀 Insider Take
Google made SERP scraping more complicated, but it did not make it impossible.
This is a good example of how the web scraping ecosystem actually works. Google changes the way its result pages are delivered, scraping providers investigate the change, patch their systems and continue returning the data their customers need.
The cost is not literally zero, but it appears to be small enough that the major providers have absorbed it so far.
The thing to monitor is not simply whether Google uses /goto links. It is whether your provider continues returning complete and accurate data across organic results, AI Overviews, local packs and other SERP features.
A 200 response does not necessarily mean you received everything you needed.
🎙️ Inside Our Latest AMAs
We recently spoke with three people working on different parts of the scraping stack. Here are the key insights that stood out.
🤖 Huey & Wade from BrowserAct: AI Isn’t Replacing Scraper Engineering
In our AMA with Huey, Product Creative Director at BrowserAct, we explored what AI-powered scraping looks like beyond the no-code demo.
Key insights from the AMA:
AI should explore once, then hand off to a reusable workflow. The strongest model is not an agent re-reasoning from scratch on every run. It is AI exploring a website, finding a workable path, then turning that discovery into a reusable Bot.
The most dangerous failures are the ones that look successful. A scraper can complete its task and return structured output while quietly extracting the wrong field, stale content or incomplete records. Production scraping still needs validation and monitoring.
🕵️ Saksham Solanki from httpcloak: Why Perfect Fingerprints Still Get Blocked
In our AMA with Saksham Solanki, creator of httpcloak, we looked at why apparently browser-identical clients still get detected.
Key insights from the AMA:
Rotating more can make you easier to detect. A real browser does not independently change its User-Agent, TLS behaviour, headers, cookies and IP. Treating each as a separate anti-blocking switch can create an identity that does not exist anywhere in the real world.
Matching Chrome once is the easy part. At production scale, request order, pacing, connection reuse, session age and connection teardown can reveal that thousands of sessions are following the same automation template.
🏠 Stan Sadokov from NodeMaven: What Proxy Providers Don’t Tell You About Residential Proxies
In our AMA with Stan Sadokov, founder of NodeMaven, we asked what actually separates a high-quality residential network from a large list of IPs.
Key insights from the AMA:
The filtering layer may matter more than the proxy pool. Providers often draw from overlapping raw supply. The real difference may be how they evaluate incoming IPs, filter poor ones, route sessions and remove addresses that have been burned.
There is no universally clean residential IP. “Clean” is not a permanent property. It is a score assigned by a particular target at a particular moment. The same IP can work on one website and fail on another.
🔎 A Few Things We Found This Month
A few interesting posts, tools, and ideas that caught our eye recently:
💸 A $1/month ScrapingBee alternative
Levelsio says he replaced a $249/month ScrapingBee plan with a Playwright scraper running on a VPS for roughly $1/month. The interesting part is the setup: real Chrome, persistent sessions, slow pacing, and better residential IPs reportedly pushed success rates to around 90%.🔍 Chrome 152 added another anti-detect signal
Chrome’s new navigator.cpuPerformance API exposes a device performance class derived largely from CPU thread count. That gives detection systems another consistency check for catching browsers that spoof hardware characteristics. Raphael Mille’s breakdown includes a prototype detector.📈 Web scraping is not dying. It became AI’s biggest client
Google searches for “web scraping” are at an all-time high, while AI labs and browser agents increasingly depend on live web data. The prediction that AI would replace scraping is looking more like AI is expanding the market for it.🧪 ShieldFont: The future weapon against AI scrapers?
ShieldFont is an open-source project that uses a custom font to show humans the original text while putting fluent decoys into the HTML that scrapers read. The idea is clever, but can it work without breaking accessibility? Screen readers depend on the same underlying page structure, creating a serious tradeoff for disabled users, even if the project provides an accessibility workaround. Ars Technica and Help Net Security have more.
That’s it for this #10 newsletter.
None of this replaces the boring work. Benchmark providers on your targets. Don’t trust a 200, a fingerprint demo, or a headline CPM. Measure cost-per-validated-payload, and keep watching whether the data you get is actually complete.
Ian from ScrapeOps
