How to use a proxy with LangChain document loaders

Your proxy details

After you buy a plan, open My proxies. Choose the Proxy Type (Rotating or Sticky), the Protocol (HTTP or SOCKS5) and a Format, then copy a line. The generator offers three formats:

In the examples on this page, replace USERNAME, PASSWORD, HOST and PORT with the values from your line. SOCKS5_PORT is the port the generator shows when you pick SOCKS5.

Residential lines are either rotating (a new IP on new connections) or sticky (one IP for the time you choose). Datacenter and ISP plans give you a list of dedicated IPs, one per line as IP:PORT:USERNAME:PASSWORD, and each line is one fixed IP.

To test a line, open https://ipinfo.io/json through it: it shows the exit IP and country. For HTTP lines you can also use the free proxy checker.

WebBaseLoader through a proxy

pip install langchain-community beautifulsoup4

from langchain_community.document_loaders import WebBaseLoader

proxy = "http://USERNAME:PASSWORD@HOST:PORT"

loader = WebBaseLoader(
    ["https://ipinfo.io/json"],
    proxies={"http": proxy, "https": proxy},
)
docs = loader.load()
print(docs[0].page_content[:300])

proxies= is applied to the loader’s own requests Session, so it affects load() and lazy_load() and nothing else in your program.

WebBaseLoader ships in langchain-community, which LangChain no longer actively maintains. It still installs and works; check LangChain’s integrations list if you move to a dedicated package.

Async loading

alazy_load() (and the older, deprecated aload()) fetch with aiohttp, which doesn’t use the proxies argument. They follow the HTTP_PROXY and HTTPS_PROXY environment variables only when you create the loader with trust_env=True.

Environment variables apply to the whole process, including the HTTP clients inside LLM SDKs. Run async loading as a separate job or process so your model API calls don’t go through the proxy and use your data.

# Run with: HTTPS_PROXY=http://USERNAME:PASSWORD@HOST:PORT python load_pages.py
import asyncio
from langchain_community.document_loaders import WebBaseLoader

urls = ["https://example.com/a", "https://example.com/b"]


async def main():
    loader = WebBaseLoader(urls, trust_env=True, requests_per_second=2)
    docs = [doc async for doc in loader.alazy_load()]
    print(len(docs))


asyncio.run(main())

Rotating or sticky?

For a RAG crawl over many sites, a rotating line spreads the load across many IPs. The loader’s Session keeps connections open per site, so pages from one site can share an IP while the connection stays open, which is fine for most public pages.

For sites that need a login or a consistent visitor, use one sticky line per loader. To make sticky lines, open the generator, set Proxy Type to Sticky, choose a Sticky Duration if your plan offers one, set how many lines you want in Sticky Amount, then copy or download them. Each line keeps its IP for the chosen time. The session part of a sticky username differs by plan, so always generate sticky lines instead of editing usernames by hand.

Target a country

On residential plans, add -country- and a two-letter country code to the end of the username, for example USERNAME-country-us or USERNAME-country-gb. To combine a country with a state, a city or a sticky session, build the line in the generator, which shows only the options your plan supports. Datacenter and ISP IPs have a fixed location that you choose when you buy them. Country targeting matters when you load pages whose content changes by region, such as prices or local news.

Pages that need JavaScript

WebBaseLoader downloads raw HTML and doesn’t run JavaScript, so single-page apps can come back nearly empty. For those, load pages with a real browser such as Playwright and pass its proxy settings there.

Tips

• Set the proxy on the loader, not with global environment variables, unless the process does nothing but load pages.
• requests_per_second caps how many pages async loading fetches at once. Keep it low on sites that block bursts.
• Datacenter IPs are enough for most documentation sites; switch to residential when you see blocks or CAPTCHAs.

Troubleshooting

• requests.exceptions.ProxyError with 407

  The proxy rejected the credentials. Check the proxy URL and URL-encode special characters.More on this error →

• alazy_load() ignores the proxy

  Async loading uses aiohttp, which doesn’t read proxies=. Use trust_env=True with environment variables in a separate process.

• Your LLM calls show up in proxy usage

  A global HTTPS_PROXY variable is routing them. Pass proxies= to the loader instead.

• Pages load empty or show a CAPTCHA

  The site needs JavaScript or blocks the IP. Use a browser loader, or residential lines.More on this error →

Seeing another error? Read proxy errors and how to fix them, or contact support.

Questions