Two years ago you could point a fleet of datacenter IPs at a news archive, run Scrapy overnight, and wake up to a clean corpus. That still works on a slice of the web. But most of the pages an AI team wants now sit behind something that reads your TLS handshake before it even glances at your headers. The data got harder to reach at exactly the moment everyone decided they needed ten times more of it.

So this is about the pipeline that survives contact with a real anti-bot stack. How the crawl and dedupe stages actually run, where proxies for AI data collection make or break the dataset, and — the part most vendor blogs skip — where a proxy is the wrong tool and you should just admit it.

Why LLM data runs hit walls classic scrapers walked past

A traditional scraper had a narrow job. Grab a product catalog, a set of listings, a price table. Low volume, predictable shape, one or two target domains. You could hand-tune it and it would hum along for months without anyone touching it.

An AI data pipeline behaves nothing like that. It wants breadth — thousands of domains, long-tail forums, regional sites, documentation, the messy middle of the web nobody bookmarks. And it wants that breadth refreshed, because a model trained on last year's snapshot answers like it's still last year. Volume times breadth times freshness is exactly the traffic profile anti-bot vendors were built to catch.

And they got good at it. Cloudflare, DataDome, Akamai, and HUMAN (the old PerimeterX) stopped caring whether you sent a browser User-Agent a long time ago. They read your JA3/JA4 TLS fingerprint. They check the ASN your IP belongs to. They watch request timing and throw challenges that need a real JavaScript runtime to clear. A datacenter IP from a well-known cloud ASN gets flagged before your request body is parsed. The block isn't personal. Your source ASN is the tell.

The old scraper's problem was parsing HTML. The AI collector's problem is proving it's a person — thousands of times an hour, from IPs that don't look like a rack in us-east-1.

That's why proxy choice went from a footnote to a load-bearing decision. Your IP is the first thing judged. On protected targets it's usually the only thing that matters.

---

The pipeline, in the order it actually runs

Every team words the stages differently, but the shape underneath is the same. Here's the sequence I'd defend in a design review:

1. Seed and discover. Start from sitemaps, curated URL lists, and a frontier queue. Cheap. Skip it and you'll spend proxy bandwidth fetching login walls and parking pages.
2. Fetch. The stage that eats proxies. Rotate egress IPs, cap per-domain concurrency, back off on 429s. Static content goes over plain HTTP clients; JS-heavy targets go through a headless browser. Residential vs. datacenter gets decided here, per-domain, not globally.
3. Render, only when forced. Playwright or a real Chromium if the content is client-rendered or gated by a JS challenge. It runs five to ten times more expensive per page than a bare fetch, so you gate it hard.
4. Extract. Pull main content out of the DOM soup — trafilatura, readability-style heuristics, or a small model for structured fields. Strip nav, ads, cookie banners, boilerplate.
5. Normalize and language-ID. Unicode cleanup, boilerplate stripping round two, detect language, drop what you can't use.
6. Dedupe. Exact hashing catches identical pages. MinHash/SimHash near-dedup catches the syndicated-article problem, where the same wire story lands on 400 sites with a different header. This stage quietly decides most of your final quality.
7. Quality filter. Heuristics plus a classifier for spam, SEO sludge, and machine-generated filler. In 2026, filtering out other models' output is a real and growing part of the job.
8. Provenance and compliance. Record source URL, fetch timestamp, and the robots/ToS status you saw at fetch time. Future-you, sitting in a legal review, will be grateful.
9. Package. Shard, compress, write to your warehouse or object store with metadata intact.

Only two stages touch proxies directly — fetch and render. But a bad proxy call poisons everything downstream. Blocked fetches become coverage gaps. Geo-blind fetches hand you the US version of a page when you needed the German one. The dataset inherits whatever your proxy layer let through, good or bad.

Freshness, coverage, geo-diversity — the three quality axes

Freshness is a recrawl-cadence problem, and cadence is capped by how many fetches survive per hour. If half your requests bounce off a challenge page, your effective refresh rate halves and your corpus drifts stale without anyone noticing.

Coverage is the long tail — the sites that block casually but persistently. A residential IP walks past most of those. A datacenter IP collects 403s.

Geo-diversity is the axis AI teams underrate. A model trained mostly on US-egress fetches learns a US-shaped web. Prices, languages, stock availability, even how an article is framed all shift by country. If you don't want a model that's quietly parochial, you fetch from the geographies you care about — which means country, and sometimes city, targeting on the proxy side. This is where residential proxies with real geo controls earn their keep.

---

Residential vs. datacenter, decided per-target

The mistake I see most is picking one proxy type for the whole crawl. You want a router, not a religion. Send each domain to the cheapest IP type that still returns a clean 200.

Factor	Datacenter	Residential
Cost	Cheapest, per proxy or flat	Higher, per-GB (roughly $2–$8/GB by volume)
Speed	Fast, low latency	Slower, varies by peer
Block rate on protected sites	High — the ASN gives it away	Low — reads as a real user
Best for	Open APIs, unprotected sites, bulk static content	DataDome/Cloudflare targets, geo-gated pages, the long tail
Geo granularity	Country-level, limited	Country + state/region + city + ISP

Rule of thumb I'll stand behind: fetch the open, unprotected 70% of your frontier over datacenter proxies because they're fast and cheap, and hold residential back for the protected minority that actually blocks you. Residential bandwidth is the expensive resource. Don't burn it pulling a plain sitemap a datacenter IP would've served for a fraction of the cost.

Now the anti-hype part. If your targets are public APIs, government open-data portals, or sites that don't fight back, residential is a waste of money. Pay for it only where the block rate earns it. One time I watched a team route their whole crawl through residential "to be safe" and torch an $1,800 bandwidth budget in a week — most of it on sites a datacenter IP would have served for pennies. A quick pass through the in-dashboard proxy tester against your real target list tells you which bucket each domain lands in before you commit a plan to it.

A rotating fetch loop that behaves

Rotation is the mechanical heart of the fetch stage. The loop below cycles egress IPs, retries on block-shaped responses, and backs off instead of hammering. It's deliberately boring. Boring is what survives a week-long crawl.

import random
import time
import httpx

# Rotating residential endpoint: the gateway hands out a fresh
# exit IP per request (or per sticky session, if you pin one).
PROXY_POOL = [
    "http://user:[email protected]:10000",
    "http://user:[email protected]:10001",
    "http://user:[email protected]:10002",
]

BLOCK_STATUS = {403, 429, 503}

def fetch(url: str, max_retries: int = 4) -> httpx.Response | None:
    for attempt in range(max_retries):
        proxy = random.choice(PROXY_POOL)
        try:
            with httpx.Client(proxy=proxy, timeout=20.0) as client:
                r = client.get(url, headers={"User-Agent": "Mozilla/5.0"})
            if r.status_code in BLOCK_STATUS:
                # Rotate IP and back off before trying again.
                time.sleep(2 ** attempt + random.random())
                continue
            r.raise_for_status()
            return r
        except (httpx.TransportError, httpx.HTTPStatusError):
            time.sleep(2 ** attempt + random.random())
    return None  # let the frontier requeue it later

if __name__ == "__main__":
    resp = fetch("https://example.com/article/123")
    print(resp.status_code if resp else "gave up")

Two notes that matter more than the code. The exponential backoff with jitter (2 ** attempt + random.random()) is what keeps a soft block from hardening into a permanent IP ban. And for anything session-bound — a login, a multi-page flow, a cart — you want a sticky session that holds one IP for its lifetime, not per-request rotation. Swapping IPs mid-session is the fastest way to trip a challenge you'd otherwise have sailed through. Rotating and sticky are both first-class on the residential and mobile pools. Pick per workload.

---

The robots/ToS gray area, said plainly

I'm not your lawyer, and this isn't legal advice. But pretending the compliance question doesn't exist is how teams end up in a cease-and-desist thread. A workable posture:

• Read robots.txt and honor it as the default. Treat overriding it as a deliberate, logged, business-signed-off exception — not something a crawler config quietly does on its own.
• Log fetch-time provenance on every record: URL, timestamp, and the robots/ToS state you saw. If a source's terms change later, you can prove what you knew when.
• Rate-limit like a guest, not a locust. Per-domain concurrency caps aren't just manners — aggressive hammering is the shortest path to a permanent block and an angry email from an ops team.
• Keep personal and copyrighted data handling inside whatever policy your legal team actually signed off on. Proxies change where you fetch from, not what you're allowed to keep.

Proxies are an access tool. They don't launder the terms of the site you're pulling from, and treating them like they do is a great way to get a whole IP range burned for everyone.

Where I'd start

Build the router first. Datacenter for the open web, residential for the protected long tail, browser-render only when a JS challenge forces your hand. Get dedupe and quality filtering right before you scale volume, because ten million junk pages is worse than one million clean ones — it just costs more to store and train on. And instrument your block rate per domain from day one. It's the single number that tells you whether your proxy layer is helping or quietly starving your dataset.

If you want to pressure-test a target list before committing to a plan, the proxy tester shows you the block picture in a few minutes, and there's more on rotation strategy and anti-bot specifics over on the blog. Cheaper to learn it there than in a stalled crawl at 3 a.m.