How to Use Proxies With Scrapy in 2026

Scrapy is one of the most widely used scraping frameworks, but running it without proxies will get your IP blocked fast, especially on sites with rate limiting or bot detection. The right way to handle that is through middleware, which manages proxy rotation at the framework level so you are not manually switching IPs on every request.

In this article, we'll explore how to set up proxy middleware in Scrapy, what your options are, and how to configure it with rotating residential proxies for the best results.

How to Set Up Proxy Middleware in Scrapy

Scrapy's built-in downloader middleware is where you plug in your proxies. Create a middleware class in your middlewares.py file:

1class ProxyMiddleware:
2    def process_request(self, request, spider):
3        request.meta['proxy'] = "http://username:password@proxy-endpoint:port"

Then enable it in settings.py:

1DOWNLOADER_MIDDLEWARES = {
2    'myproject.middlewares.ProxyMiddleware': 350,
3}

If you are using a provider like Proxyon, you only need one endpoint URL. IP rotation is handled on their end automatically, so every request goes out through a different IP with no extra logic on your side.

If you want to rotate through a list of proxies yourself, extend the middleware to pick a random one per request:

1import random
2
3class ProxyMiddleware:
4    proxies = [
5        "http://user:pass@proxy1:port",
6        "http://user:pass@proxy2:port",
7    ]
8
9    def process_request(self, request, spider):
10        request.meta['proxy'] = random.choice(self.proxies)

Managing a proxy list manually gets messy fast. A single rotating endpoint is the cleaner approach for anything beyond basic testing.

Also Read: Best Residential Proxy Providers

Residential vs Datacenter for Scrapy

Datacenter proxies are fast and cheap, but they come from commercial server ranges that websites can easily flag. If your target has serious bot detection, you will hit blocks quickly. Residential proxies use IPs assigned to real households by ISPs, so they look like normal user traffic and are much harder to block on protected targets like e-commerce sites or anything behind Cloudflare.

Start with datacenter proxies if your target is unprotected and cost matters. Switch to residential when you start seeing 403s or CAPTCHAs. The middleware setup stays exactly the same either way; you just swap the credentials. Proxyon offers both with no subscription required.

Keeping Your Scraper Running

Pace your requests. Even with rotating proxies, sending requests too fast will trigger rate limiting. Add a download delay in your settings.py:

1DOWNLOAD_DELAY = 1

Also, make sure retry handling is enabled. If a proxy returns an error, Scrapy will fail that request unless you tell it otherwise. Set RETRY_ENABLED = True and configure a retry count so failed requests automatically cycle through a new IP instead of dying silently.

The last thing is matching the right proxy type to your target. Datacenter proxies on a heavily protected site will get you blocked, no matter how well your middleware is set up. If you are seeing consistent blocks, switching to residential proxies is usually the fix.

Also Read: What is Proxyon?

Final Thoughts

A few lines of middleware, the right proxy type for your target, a request delay, and retry handling, and your Scrapy scraper will run without interruptions. Proxyon's residential proxies start at $1.75/GB with no subscription required. Deposit $5 and start scraping in minutes at Proxyon.