Slow performance

Measure where the time goes

cURL can break a request into its stages. Run this through your proxy:

curl -s -o /dev/null -x http://USERNAME:PASSWORD@HOST:PORT \
  -w "connect: %{time_connect}s\ntunnel:  %{time_appconnect}s\nfirst byte: %{time_starttransfer}s\ntotal:   %{time_total}s\n" \
  https://api.ipify.org

• connect is the time to reach the proxy from your machine.
• tunnel adds the hop from the proxy to the site and the TLS handshake.
• first byte adds the time the site takes to start replying.

Pine’s network typically answers in under 50 ms on US connections. If connect is high, the delay is between you and the proxy. If first byte jumps after tunnel, the site is slow to reply.

Compare with and without the proxy

Run the same command against your target site, once with -x and once without. If both are slow, the site is slow, and the proxy isn’t the cause.

Reuse connections

A new connection costs a TCP handshake to the proxy and a TLS handshake to the site, every time. Keep connections open between requests:

• Python requests: use a requests.Session() instead of calling requests.get() each time.
• Node.js: reuse one ProxyAgent for all requests, instead of creating one per request.
• Scrapy and browsers: both reuse connections already. Don’t restart them between pages.

Spread the load across your IPs

Hundreds of parallel requests through one IP queue up behind each other, and sites notice the burst. Spread concurrent requests across all of your IPs, and start at about 5 to 10 at once per IP. Python requests and Scrapy both show how.

Cut what you download

Bandwidth is unlimited and never throttled, but smaller replies still arrive sooner.

• Send Accept-Encoding: gzip, br so sites compress their replies. Most HTTP clients do this already.
• In headless browsers, skip images, fonts and media you don’t need. In Puppeteer, use request interception to abort them.
• Request an API or JSON endpoint instead of the full page, when the site offers one.

Distance

Pine IPs are in the United States, so sites hosted in the US answer fastest. Sites in Europe or Asia add the round trip across an ocean to every request. Reusing connections, as shown above, matters most there.

Set sensible timeouts

A few very slow requests can stall a whole crawl. Use a short connect timeout and a longer read timeout, such as 5 seconds to connect and 30 seconds to wait for the reply. Then retry anything that times out.