Every HTTP request carries a user agent string identifying the browser and OS that sent it. Skip this header in Python's Requests library, and the default value flags your script as a bot before anything else gets checked. Rotating through real browser user agents makes each request look like a different visitor.
In this article, we'll explore how user agents get scrapers blocked, how to rotate them in Python, and how to build a list that stays current.
Why Your Scraper's User Agent Gets You Blocked
Python's Requests library sends a default user agent like python-requests/2.31.0 when you don't set one, telling the server exactly what made the request. Most sites log this string for bot detection, and a default or missing value is one of the easiest signals to check.
A single fixed "real" user agent isn't much better either. If every request carries the same string, the pattern becomes obvious at scale. The fix is rotating through a pool of genuine user agents, so each request looks like a different visitor.
Also Read: What Are Rotating Proxies?
How to Rotate User Agents in Python
Keep a list of real browser user agents and pick one at random for each request with Python's random module. Full syntax is in the Requests documentation.
import requests
import random
user_agents = [
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4 Safari/605.1.15",
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36",
"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:125.0) Gecko/20100101 Firefox/125.0",
]
url = "https://example.com"
for i in range(5):
headers = {"User-Agent": random.choice(user_agents)\}
response = requests.get(url, headers=headers)
print(f"Request {i+1\}: status {response.status_code\}")Each request now sends a different header, so traffic looks like it's coming from a different browser. Pair this with rotating residential proxies for the best results, since a fresh IP and user agent together are much harder to fingerprint. Run a mismatched proxy list through Proxy Formatter first to get the IP: PORT pairs your script expects.
Building a Reliable User Agent List
A hardcoded list goes stale fast. Chrome and Firefox ship new releases every few weeks, and a two-year-old version string is almost as suspicious as no user agent at all.
Pull strings from a source that tracks real browser usage, like WhatIsMyBrowser's user agent list, which shows active versions across Windows, macOS, and Linux. Refresh every month or two. Ten to twenty varied entries covering different browsers and systems are enough, mix in mobile if your target serves it, and drop anything outdated.
Also Read: How to Scrape Ebay Listings
Final Thoughts
Rotating user agents removes one of the easiest signals sites use to spot bots. Combine it with rotating proxies and a refreshed list, and your scraper blends in with normal traffic.
