How to use proxies with Crawlee

Your proxy details

After you buy a plan, open My proxies. Choose the Proxy Type (Rotating or Sticky), the Protocol (HTTP or SOCKS5) and a Format, then copy a line. The generator offers three formats:

In the examples on this page, replace USERNAME, PASSWORD, HOST and PORT with the values from your line. SOCKS5_PORT is the port the generator shows when you pick SOCKS5.

Residential lines are either rotating (a new IP on new connections) or sticky (one IP for the time you choose). Datacenter and ISP plans give you a list of dedicated IPs, one per line as IP:PORT:USERNAME:PASSWORD, and each line is one fixed IP.

To test a line, open https://ipinfo.io/json through it: it shows the exit IP and country. For HTTP lines you can also use the free proxy checker.

A rotating line

npm install crawlee

import { CheerioCrawler, ProxyConfiguration } from 'crawlee';

const proxyConfiguration = new ProxyConfiguration({
  proxyUrls: ['http://USERNAME:PASSWORD@HOST:PORT'],
});

const crawler = new CheerioCrawler({
  proxyConfiguration,
  async requestHandler({ request, body, log }) {
    log.info(request.url + ' -> ' + body.toString());
  },
});

await crawler.run(['https://ipinfo.io/json']);

The same proxyConfiguration option works with HttpCrawler, JSDOMCrawler, PlaywrightCrawler and PuppeteerCrawler. Browser crawlers handle the proxy username and password for you.

Sticky IPs per session

With a session pool, Crawlee keeps each session on one proxy URL, and it retires sessions that get blocked. Give it a list of sticky lines and each session keeps its own IP, cookies and all.

To make sticky lines, open the generator, set Proxy Type to Sticky, choose a Sticky Duration if your plan offers one, set how many lines you want in Sticky Amount, then copy or download them. Each line keeps its IP for the chosen time. The session part of a sticky username differs by plan, so always generate sticky lines instead of editing usernames by hand.

import { readFileSync } from 'node:fs';
import { PlaywrightCrawler, ProxyConfiguration } from 'crawlee';

// sticky.txt: sticky lines from the generator, Format: protocol://username:password@hostname:port
const proxyUrls = readFileSync('sticky.txt', 'utf8')
  .split('\n')
  .map((line) => line.trim())
  .filter(Boolean);

const crawler = new PlaywrightCrawler({
  proxyConfiguration: new ProxyConfiguration({ proxyUrls }),
  useSessionPool: true,
  persistCookiesPerSession: true,
  async requestHandler({ page, session, log }) {
    log.info('session ' + session?.id + ': ' + (await page.textContent('body')));
  },
});

await crawler.run(['https://ipinfo.io/json']);

Target a country

On residential plans, add -country- and a two-letter country code to the end of the username, for example USERNAME-country-us or USERNAME-country-gb. To combine a country with a state, a city or a sticky session, build the line in the generator, which shows only the options your plan supports. Datacenter and ISP IPs have a fixed location that you choose when you buy them.

new ProxyConfiguration({ proxyUrls: ['http://USERNAME-country-us:PASSWORD@HOST:PORT'] });

Only use the proxy when you need it

tieredProxyUrls starts without a proxy (or on cheaper IPs) and moves up a tier when Crawlee detects blocking. It saves data on sites that don’t block you.

const proxyConfiguration = new ProxyConfiguration({
  tieredProxyUrls: [
    [null],                                   // first try without a proxy
    ['http://USERNAME:PASSWORD@HOST:PORT'],   // then through ZenProxy
  ],
});

Troubleshooting

• 407 in the request log

  The proxy rejected the credentials. Check the URL, including the http:// prefix.More on this error →

• Content-Type errors on files such as PDFs

  HTTP crawlers accept a set list of content types. Add the type to the crawler’s additionalMimeTypes option.

• Sessions keep getting retired

  The site blocks the IPs or the pace. Lower maxConcurrency, and use residential lines.More on this error →

• Timeouts

  Set an explicit timeout of 30–60 seconds, retry on a new connection, and lower concurrency.More on this error →

Seeing another error? Read proxy errors and how to fix them, or contact support.

Questions