Scrapy

One request

Scrapy’s built-in HttpProxyMiddleware is on by default. Set proxy in a request’s meta, with the login in the URL, and Scrapy sends the credentials for you:

import scrapy

class BooksSpider(scrapy.Spider):
    name = "books"

    def start_requests(self):
        yield scrapy.Request(
            "https://books.toscrape.com/",
            meta={"proxy": "http://USERNAME:PASSWORD@HOST:PORT"},
        )

    def parse(self, response):
        for title in response.css("article.product_pod h3 a::attr(title)").getall():
            yield {"title": title}

Every request, across all your IPs

To proxy the whole crawl, add a small downloader middleware that sets proxy on each request. This one cycles through your IPs:

# myproject/middlewares.py
import itertools

class PineProxyMiddleware:
    def __init__(self, hosts, username, password):
        urls = [f"http://{username}:{password}@{host}" for host in hosts]
        self.pool = itertools.cycle(urls)

    @classmethod
    def from_crawler(cls, crawler):
        settings = crawler.settings
        return cls(
            settings.getlist("PINE_HOSTS"),
            settings.get("PINE_USERNAME"),
            settings.get("PINE_PASSWORD"),
        )

    def process_request(self, request, spider):
        request.meta.setdefault("proxy", next(self.pool))

Then turn it on in settings.py. Keep the password out of source control by reading it from the environment:

# settings.py
import os

PINE_HOSTS = ["HOST_1:PORT", "HOST_2:PORT", "HOST_3:PORT"]
PINE_USERNAME = os.environ["PINE_USERNAME"]
PINE_PASSWORD = os.environ["PINE_PASSWORD"]

DOWNLOADER_MIDDLEWARES = {
    # Runs before HttpProxyMiddleware (750), which reads meta["proxy"].
    "myproject.middlewares.PineProxyMiddleware": 350,
}

Using setdefault means a request that already names a proxy keeps it. That’s handy for pinning one site’s logged-in session to a single static IP.

Pace the crawl

Bandwidth is unlimited, so the limit worth setting is the one the site will tolerate. Keep concurrency per site modest and let Scrapy slow down when the site does:

# settings.py
CONCURRENT_REQUESTS = 32
CONCURRENT_REQUESTS_PER_DOMAIN = 8
AUTOTHROTTLE_ENABLED = True
AUTOTHROTTLE_TARGET_CONCURRENCY = 4
RETRY_HTTP_CODES = [429, 500, 502, 503, 504]

Pick a fingerprint

Put the profile suffix in the username, and send a User-Agent for the same OS:

# settings.py
PINE_USERNAME = os.environ["PINE_USERNAME"] + "-os-macos"
USER_AGENT = (
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 14_5) AppleWebKit/605.1.15 "
    "(KHTML, like Gecko) Version/17.5 Safari/605.1.15"
)

Check it works

This spider logs the IP each request leaves from:

import scrapy

class IPCheckSpider(scrapy.Spider):
    name = "ipcheck"
    start_urls = ["https://api.ipify.org?format=json"] * 3

    def start_requests(self):
        for url in self.start_urls:
            yield scrapy.Request(url, dont_filter=True)

    def parse(self, response):
        self.logger.info("Exit IP: %s", response.json()["ip"])

Run it:

scrapy crawl ipcheck

With the middleware above, the three lines show three of your Pine IPs.

SOCKS5

Scrapy only supports HTTP proxies. That covers both http:// and https:// sites, so you don’t need SOCKS5 for a crawl. If you need it for something else, put a local HTTP-to-SOCKS bridge in front, or use a client that supports SOCKS5 directly, such as requests.