Plug Proxyon Into Scrapy

Route your Scrapy spiders through Proxyon in a few lines of middleware config, no extra packages required.

TL;DR: Scrapy's built-in HttpProxyMiddleware handles proxy auth for you. Set your Proxyon endpoint in settings.py or request.meta, raise RETRY_TIMES, and your spider routes through rotating IPs with almost no code.

Set the Proxy

Add your Proxyon credentials directly in settings.py for global use:

HTTPPROXY_ENABLED = True
DOWNLOADER_MIDDLEWARES = {
    'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': 110,
}
HTTP_PROXY = 'http://username:password@proxy.proxyon.io:port'

For per-request control, set it in the spider instead:

yield scrapy.Request(
    url='https://example.com',
    meta={'proxy': 'http://username:password@proxy.proxyon.io:port'}
)

The middleware handles the Proxy-Authorization header automatically once credentials are in the URL.

Rotate Proxies Without Touching Spider Logic

A custom downloader middleware is the cleanest way to rotate IPs per request:

class ProxyonMiddleware:
    def process_request(self, request, spider):
        request.meta['proxy'] = 'http://username:password@proxy.proxyon.io:port'

DOWNLOADER_MIDDLEWARES = {
    'myproject.middlewares.ProxyonMiddleware': 100,
}

With residential proxies, rotation happens on the backend through a single endpoint. No proxy list to maintain, no manual switching.

Also Read: How to Scrape a Site That Requires a Login

Retries and Proxy Choice

Proxy requests fail more than direct ones. Raise your retry tolerance so one bad IP doesn't kill a page:

RETRY_TIMES = 5
RETRY_HTTP_CODES = [500, 502, 503, 504, 408, 429]

For lightly protected targets, datacenter proxies are faster and cheaper. For sites with real bot detection, residential IPs get through more consistently. Match the proxy to the target, not habit.

Also Read: How to Scrape Google for SEO Data With Proxies

FAQ Section

Does Scrapy support SOCKS5?

Not natively. You'll need scrapy-socks5 or a custom handler, since the built-in middleware only speaks HTTP-style proxy auth.

Can I rotate per request instead of per session?

Yes, generate a new proxy URL inside process_request for each call.

Why am I still getting blocked with a proxy set?

Usually pacing, not the proxy. Slow down concurrency and randomize delays alongside rotation.

Do I need a paid proxy plan to test my Scrapy setup?

No. You can test the middleware wiring with a single static proxy before scaling to rotation.

Does the proxy middleware slow down my spider?

Slightly, since traffic takes an extra hop. The tradeoff is worth it once you're hitting sites that block direct requests.

Can I use different proxy types for different spiders in the same project?

Yes. Set the proxy per spider in custom_settings or override it in start_requests, so each spider picks the proxy type that fits its target.

Final Thoughts

Scrapy's proxy handling stays out of your way once the middleware is wired in correctly. Proxyon plugs into that setup with pay-as-you-go access and no subscription to manage. Deposit and start scraping at proxyon.io.