Python Web Scraping Tutorial (2026)

Python is one of the most popular languages for web scraping, and for good reason; it has mature libraries, a straightforward syntax, and a large community that has already solved most of the problems you will run into. If you need to pull product prices, monitor content changes, or collect data at scale, Python gives you the tools to do it without overcomplicating things.

In this article, we'll explore how to scrape the web with Python from scratch, step by step.

Installing the Right Libraries

You only need two libraries to get started. Requests handles sending HTTP requests to a website and getting the HTML back. BeautifulSoup parses that HTML so you can pull out exactly what you need. Install both with a single command:

1pip install requests beautifulsoup4

If the site you are scraping renders content with JavaScript, neither of these will be enough on their own. In that case, you need Playwright or Selenium, which control a real browser and wait for the page to fully load before pulling the data. Install Playwright with:

1pip install playwright
2playwright install

For most static sites, requests and BeautifulSoup are all you need. Only bring in a browser automation library when the target site actually requires it, since it adds overhead and slows things down.

Extracting the Data

Once you have the HTML, extracting data comes down to identifying the right elements. BeautifulSoup lets you search by tag, class, or ID. Here is a basic example that fetches a page and pulls all the links:

1import requests
2from bs4 import BeautifulSoup
3
4response = requests.get("https://example.com")
5soup = BeautifulSoup(response.text, "html.parser")
6
7links = soup.find_all("a")
8for link in links:
9    print(link.get("href"))

If you need something more specific, use find() to grab a single element or target it by class name:

1title = soup.find("h1", class_="product-title")
2print(title.text)

For more complex pages, browser developer tools are your best friend. Right-click any element, hit Inspect, and you can see exactly what tag and class to target in your code. Once you have the data extracted, you can store it in a CSV or push it directly into a database, depending on what your use case requires.

Avoiding Blocks and Errors

Most websites will block you if your requests come in too fast or look like they are coming from a bot. The first thing to do is add a delay between requests using time.sleep(), even a one or two-second pause, reduces the chance of getting flagged significantly.

The second thing is to set a User-Agent header so your requests look like they are coming from a real browser:

1headers = {"User-Agent": "Mozilla/5.0"}
2response = requests.get("https://example.com", headers=headers)

If you are scraping at scale, rotating proxies are the most reliable way to avoid IP bans. Instead of all your requests coming from one address, each request goes out through a different IP, making it look like normal traffic from multiple users. Proxyon offers rotating residential proxies starting at $1.75/GB with no subscription, which is a straightforward option if you need to scrape without interruptions.

Wrap your requests in a try/except block to handle errors cleanly without crashing your script:

1try:
2    response = requests.get("https://example.com", headers=headers)
3    response.raise_for_status()
4except requests.exceptions.RequestException as e:
5    print(f"Request failed: {e}")

These three things, pacing your requests, setting headers, and rotating proxies, cover the majority of blocking issues you will run into.

Also Read: How to Rotate Proxies in Python Requests

Final Thoughts

Get the libraries set up, write clean extraction logic, and handle your requests properly. Start simple with requests and BeautifulSoup, add Playwright when the site needs it, and use rotating proxies when you are scraping at scale. That covers everything you need to get a scraper running.