Blocked requests

Why sites block requests

A residential IP clears the checks that reject datacenter ranges outright. Sites then look at how the request behaves, and that’s where most blocks come from:

Signal	What gives you away	Fix
Pace	A burst of requests from one address	Slow down
Headers	The short header set an HTTP library sends by default	Look like the browser you claim
TCP/IP fingerprint	A handshake from a different OS than the User-Agent names	Look like the browser you claim
TLS fingerprint	A handshake shaped like a script, not a browser	Use a real browser
Challenges	Captchas and JavaScript checks	Use a real browser

Start at the top of the table. Pace and headers explain most blocks, and they’re the cheapest to fix.

Look like the browser you claim

The most realistic headers are the ones your own browser sends. Open the site in it, then in the developer tools’ Network tab, pick the page request and copy its request headers. Keep the set together: a Safari User-Agent next to headers only Chrome sends is a giveaway of its own.

Then pick the TCP/IP profile for the same system. This session claims to be Safari on a Mac, so it connects with the macOS profile:

import requests

session = requests.Session()
session.headers.update({
    # Copied from Safari on macOS. Replace with your own browser's set.
    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 14_5) AppleWebKit/605.1.15 "
                  "(KHTML, like Gecko) Version/17.5 Safari/605.1.15",
    "Accept-Language": "en-US",
})
proxy = "http://USERNAME-os-macos:PASSWORD@HOST:PORT"
session.proxies = {"http": proxy, "https": proxy}

With no suffix, Pine connects as Windows 11, which suits a Windows User-Agent. TCP/IP fingerprints lists the suffix for each system.

Slow down

Your IPs are static, and a site remembers an address that crawled too hard. Pacing your requests works better than switching IPs once they’re blocked:

import random
import time

for url in urls:
    response = session.get(url, timeout=20)
    time.sleep(random.uniform(1, 3))  # a random 1 to 3 second pause

Spread the work across all your IPs as well, so each one stays at a pace a person might keep. In Scrapy, AUTOTHROTTLE_ENABLED = True does this for you. See Scrapy.

Back off when asked

A 429 means the site wants you to slow down, and many include a Retry-After header that says for how long. Wait that long, then wait longer after each repeat:

import time

def fetch(session, url, attempts=4):
    for attempt in range(attempts):
        response = session.get(url, timeout=20)
        if response.status_code != 429:
            return response
        wait = int(response.headers.get("Retry-After", 5 * 2 ** attempt))
        time.sleep(wait)
    return response

Use a real browser for hard sites

Some sites check JavaScript, cookies and TLS details that only a real browser gets right. For those, use Puppeteer or Playwright through your Pine IPs, rather than a plain HTTP client.

What the status code tells you

Code	Meaning	What to do
403	The site refused the request	Fix headers and the fingerprint, slow down, or use a browser
429	Too many requests	Back off and follow Retry-After
503	The site is busy, or showing a challenge	Retry later, or use a browser if it shows a challenge page
407	The proxy didn’t accept the login	Check your credentials. See Connection errors.