Most scraping tutorials assume the page returns HTML. JavaScript-heavy sites don't. The content renders after the browser executes JS, which means requests get you an empty shell. Playwright fixes that. It runs a real browser. Pair it with rotating residential proxies, and you can scrape sites that block every other approach.
Why Playwright Over Requests
Playwright controls a full Chromium, Firefox, or WebKit browser. It waits for JS to execute, handles dynamic content loading, and lets you interact with the page the same way a real user would. The downside is speed. Browser automation is slower and heavier than plain HTTP. Use it only when the target site actually requires it.
Setting Up Playwright With a Proxy
I Playwright and its browser binaries first:
1-pip install playwright
2-Playwright installs Chromium
3-Proxy configuration goes at the browser context level, not per-request. This is the key difference from requests:
1from playwright.sync_api import sync_playwright
2
3PROXY = {
4 "server": "http://residential.proxyon.io:8080",
5 "username": "YOUR_USERNAME",
6 "password": "YOUR_PASSWORD"
7}
8with sync_playwright() as p:
9 browser = p.chromium.launch()
10 context = browser.new_context(proxy=PROXY)
11 page = context.new_page()
12 page.goto("https://example.com")
13 print(page.content())
14 browser.close()Setting the proxy on the context means every page opened from that context routes through the same IP. Create a new context per session when you need a fresh IP.
Rotating IPs Between Sessions
For large-scale web scraping, create a new browser context per URL and pass a different proxy each time:
from playwright.sync_api import sync_playwright
1import requests
2proxies = {
3 "http": "http://user:[email protected]:8000",
4 "https": "http://user:[email protected]:8000"
5}
6response = requests.get("https://target.com", proxies=proxies)
7
8
9Setup takes under 60 seconds with instant account activation. Developers get APIwait_until="networkidle" tells Playwright to wait until no network requests have fired for 500ms, which is useful for pages that load content in multiple async waves.
Also Read: How to Do Web Scraping Without Getting Blocked (2026)
Bypassing Cloudflare With Residential IPs
Cloudflare's bot detection scores traffic based on IP reputation, browser fingerprint, and behavioral patterns. Datacenter IPs fail this check almost immediately. Residential proxies come from real ISP-assigned addresses, which pass the reputation check by default.
Two things help beyond the IP itself. First, set a real user agent in the context:
1context = browser.new_context(
2 proxy=PROXY,
3 user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"
4)Second, avoid headless=True on heavily protected targets. Cloudflare detects the headless flag. Use channel="chrome" with Playwright's persistent context instead, which mimics a real Chrome installation more closely.
Final Thoughts
Launching a new browser instance per request is the fastest way to kill your scraper's performance. Launch once, open new contexts per session. Browser startup is expensive, context creation is cheap.
Skipping wait_until on JS-heavy pages means you'll capture the page before the content renders. Always specify "networkidle" or "domcontentloaded" depending on how the target loads its data.
Playwright with rotating residential proxies handles the sites that break every other scraping approach. The setup takes about 20 lines. Proxyon's residential proxies start at $1.75/GB with no subscription required. Deposit $5 and start scraping in minutes at proxyon.io.
