How to use rotating proxies with Scrapy

Your proxy details

After you buy a plan, open My proxies. Choose the Proxy Type (Rotating or Sticky), the Protocol (HTTP or SOCKS5) and a Format, then copy a line. The generator offers three formats:

In the examples on this page, replace USERNAME, PASSWORD, HOST and PORT with the values from your line. SOCKS5_PORT is the port the generator shows when you pick SOCKS5.

Residential lines are either rotating (a new IP on new connections) or sticky (one IP for the time you choose). Datacenter and ISP plans give you a list of dedicated IPs, one per line as IP:PORT:USERNAME:PASSWORD, and each line is one fixed IP.

To test a line, open https://ipinfo.io/json through it: it shows the exit IP and country. For HTTP lines you can also use the free proxy checker.

Set the proxy on a request

import scrapy

PROXY = "http://USERNAME:PASSWORD@HOST:PORT"


class IpSpider(scrapy.Spider):
    name = "ip"

    async def start(self):  # Scrapy 2.13 and later
        for request in self.start_requests():
            yield request

    def start_requests(self):  # older Scrapy versions
        yield scrapy.Request("https://ipinfo.io/json", meta={"proxy": PROXY})

    def parse(self, response):
        yield response.json()

Scrapy turns the username and password in the URL into a Proxy-Authorization header. A proxy in meta takes priority over the http_proxy and https_proxy environment variables, which Scrapy also reads.

Scrapy 2.13 introduced async def start() for start requests, and recent versions no longer call start_requests() on their own. The spider above defines both, so it runs on old and new versions.

Use the proxy for every request

A small downloader middleware sets the proxy on each request. Give it a number below 750 so it runs before the built-in HttpProxyMiddleware. The spider=None default keeps the method working on recent Scrapy versions, which stop passing the spider argument.

class ZenProxyMiddleware:
    proxy = "http://USERNAME:PASSWORD@HOST:PORT"

    def process_request(self, request, spider=None):
        request.meta.setdefault("proxy", self.proxy)

DOWNLOADER_MIDDLEWARES = {
    "myproject.middlewares.ZenProxyMiddleware": 350,
}

How rotation works in Scrapy

Rotation happens per connection. A rotating line gets a new IP when your client opens a new connection; requests sent through the same open HTTPS tunnel leave from the same IP. Scrapy keeps a pool of open connections per site and reuses them, so with one rotating line you see a handful of IPs at a time rather than a new one on every request.

When you need a different IP per request, give Scrapy many lines and pick one at random for each request. Generate them as sticky lines (set Sticky Amount to, say, 100) or use your datacenter or ISP IP list. The middleware below also picks a new line when Scrapy retries a request.

import random
from urllib.parse import quote


def to_proxy_url(line):
    line = line.strip()
    if "://" in line:
        return line
    if "@" in line:
        return "http://" + line
    host, port, user, password = line.split(":", 3)
    return "http://%s:%s@%s:%s" % (quote(user, safe=""), quote(password, safe=""), host, port)


class ProxyListMiddleware:
    def __init__(self, proxies):
        self.proxies = proxies

    @classmethod
    def from_crawler(cls, crawler):
        path = crawler.settings.get("PROXY_LIST_FILE", "proxies.txt")
        with open(path) as f:
            proxies = [to_proxy_url(line) for line in f if line.strip()]
        return cls(proxies)

    def process_request(self, request, spider=None):
        # New line on first try and on every retry
        if "proxy" not in request.meta or request.meta.get("retry_times"):
            request.meta["proxy"] = random.choice(self.proxies)

DOWNLOADER_MIDDLEWARES = {
    "myproject.middlewares.ProxyListMiddleware": 350,
}
PROXY_LIST_FILE = "proxies.txt"  # one line per proxy, any generator format

Target a country

On residential plans, add -country- and a two-letter country code to the end of the username, for example USERNAME-country-us or USERNAME-country-gb. To combine a country with a state, a city or a sticky session, build the line in the generator, which shows only the options your plan supports. Datacenter and ISP IPs have a fixed location that you choose when you buy them.

yield scrapy.Request(url, meta={"proxy": "http://USERNAME-country-us:PASSWORD@HOST:PORT"})

Settings that help on protected sites

CONCURRENT_REQUESTS_PER_DOMAIN = 4
DOWNLOAD_TIMEOUT = 60          # the default is 180 seconds
RETRY_TIMES = 3
AUTOTHROTTLE_ENABLED = True

Scrapy retries 429 and common 5xx responses by default. With the proxy-list middleware above, each retry goes out through a different line.

SOCKS5

Scrapy’s default download handlers don’t support SOCKS proxies. Scrapy 2.15 added an experimental httpx-based handler that does, but it isn’t recommended for production yet, so use HTTP lines with Scrapy.

Troubleshooting

• TunnelError: Could not open CONNECT tunnel … 407

  The proxy rejected the credentials. Check the line and URL-encode special characters.More on this error →

• TunnelError with 502 or 503

  The proxy couldn’t reach the site through the exit IP. Let Scrapy retry on a new line.More on this error →

• Your middleware doesn’t seem to run

  Check the module path in DOWNLOADER_MIDDLEWARES and keep its number below 750.

• The IP doesn’t change on a rotating line

  Your client is reusing an open connection. Requests that share a tunnel share its IP. Open a new connection per request, or use one sticky line per task.More on this error →

• 403 or 429 from the site

  Lower concurrency, enable AutoThrottle and spread requests over more lines.More on this error →

Seeing another error? Read proxy errors and how to fix them, or contact support.

Questions