Python requests

Basic request

Pass the proxy URL for both http and https. Both keys use the http:// scheme: your client talks plain HTTP to the proxy and opens an encrypted tunnel through it for https:// sites.

import requests

proxy = "http://USERNAME:PASSWORD@HOST:PORT"
proxies = {"http": proxy, "https": proxy}

response = requests.get("https://api.ipify.org?format=json", proxies=proxies, timeout=20)
print(response.json())  # {"ip": "<your Pine IP>"}

If you leave out the https key, https:// requests skip the proxy and leave from your own IP.

Reuse connections with a session

A Session keeps connections open between requests, which saves a TCP and TLS handshake on every call after the first:

import requests

session = requests.Session()
session.proxies = {
    "http": "http://USERNAME:PASSWORD@HOST:PORT",
    "https": "http://USERNAME:PASSWORD@HOST:PORT",
}

for page in range(1, 4):
    response = session.get(f"https://books.toscrape.com/catalogue/page-{page}.html", timeout=20)
    print(page, response.status_code)

Environment variables such as HTTPS_PROXY override session.proxies when trust_env is on, which it is by default. Set session.trust_env = False if a stray variable is sending traffic elsewhere.

Spread requests across your IPs

Your Pine IPs share one username and password, so each proxy URL differs only by host and port. Cycle through them to spread the load:

import itertools
import requests

hosts = ["HOST_1:PORT", "HOST_2:PORT", "HOST_3:PORT"]
pool = itertools.cycle(f"http://USERNAME:PASSWORD@{host}" for host in hosts)

def fetch(url):
    proxy = next(pool)
    return requests.get(url, proxies={"http": proxy, "https": proxy}, timeout=20)

print(fetch("https://api.ipify.org").text)

Every IP is static, so the same host always gives you the same exit address. To keep one site’s session on one address, give that site a fixed host instead of cycling.

SOCKS5

Install the SOCKS extra, then use the socks5h:// scheme so DNS lookups happen at the proxy:

pip install "requests[socks]"

Then point both keys at the SOCKS5 URL:

proxy = "socks5h://USERNAME:PASSWORD@HOST:PORT"
response = requests.get("https://api.ipify.org", proxies={"http": proxy, "https": proxy}, timeout=20)

With socks5:// (no h), your machine resolves the hostname, and the lookup leaves from your own network.

Pick a fingerprint

Add a suffix to the username to send a different TCP/IP fingerprint. Match it to your User-Agent:

proxy = "http://USERNAME-os-macos:PASSWORD@HOST:PORT"
headers = {
    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 14_5) AppleWebKit/605.1.15 "
                  "(KHTML, like Gecko) Version/17.5 Safari/605.1.15",
}
response = requests.get("https://api.ipify.org", proxies={"http": proxy, "https": proxy}, headers=headers, timeout=20)

TCP/IP fingerprints lists every profile.

Timeouts, errors and retries

Always set a timeout. Without one, a request to a slow site can hang forever. Retry the errors that clear up on their own, and back off when a site asks you to:

import requests
from requests.adapters import HTTPAdapter
from urllib3.util import Retry

retry = Retry(
    total=3,
    backoff_factor=1,                    # waits 1s, 2s, then 4s
    status_forcelist=[429, 500, 502, 503, 504],
    respect_retry_after_header=True,
)
session = requests.Session()
session.mount("http://", HTTPAdapter(max_retries=retry))
session.mount("https://", HTTPAdapter(max_retries=retry))
session.proxies = {"http": "http://USERNAME:PASSWORD@HOST:PORT", "https": "http://USERNAME:PASSWORD@HOST:PORT"}

try:
    response = session.get("https://api.ipify.org", timeout=(5, 30))
    response.raise_for_status()
except requests.exceptions.ProxyError:
    print("Couldn't reach the proxy, or it rejected the login.")
except requests.exceptions.Timeout:
    print("The request timed out.")

The timeout pair is (connect, read): five seconds to connect, thirty to wait for the reply.