Most people pulling data from websites end up copying it manually, which works until the dataset gets too large or the site updates. Scraping directly into Excel removes that bottleneck by automating the collection and dropping the output exactly where you need it, ready for analysis. Whether you are working with product prices, leads, or any structured data, the process follows the same basic pattern.
In this article, we'll explore how to scrape website data straight into Excel using Python.
How to Scrape Data with Python
You need three libraries to get started: Requests to fetch the page, BeautifulSoup to parse the HTML, and Pandas to structure and export the data. Install them with:
pip install requests beautifulsoup4 pandas openpyxl
Once installed, the basic scraper looks like this:
import requests
from bs4 import BeautifulSoup
import pandas as pd
url = 'https://example.com/products'
response = requests.get(url)
soup = BeautifulSoup(response.text, 'html.parser')
data = []
for item in soup.find_all('td'):
data.append(item.get_text())
df = pd.DataFrame(data, columns=['Column1'])
df.to_excel('output.xlsx', index=False)$Requests sends a GET request and pulls the raw HTML. BeautifulSoup then parses it so you can locate specific elements using tag names or class attributes. Adjust find_all() to match your target, whether that is table rows, product names, or prices. Before running your scraper, you can verify your proxy connection using Proxyon's free proxy tools.
Also Read: How to Scrape Ebay Listings
How to Export Scraped Data to Excel
Store each scraped record as a dictionary so the keys become column headers automatically. Then pass the list into Pandas and write it to Excel:
data = []
for book in books:
data.append({
"Title": book.h3.a["title"],
"Price": book.find("p", class_="price_color").text
\})
df = pd.DataFrame(data)
df.to_excel("output.xlsx", index=False, engine="openpyxl")$If you are scraping multiple categories, ExcelWriter lets you write each dataset to its own sheet within a single workbook:
with pd.ExcelWriter("output.xlsx", engine="openpyxl") as writer:
df_products.to_excel(writer, sheet_name="Products", index=False)
df_prices.to_excel(writer, sheet_name="Prices", index=False)$The .xlsx file appears in the same directory as your script once the code runs.
How to Handle Dynamic and Protected Sites
Requests and BeautifulSoup only work on pages that return HTML directly. Sites built on frameworks like React or Vue render content through JavaScript after the initial load, so your scraper gets an empty shell. For those, you need Playwright, which runs a real browser and waits for JavaScript to execute before returning the page content.
For sites that block scrapers by IP, rotating residential proxies keeps your scraper running by cycling through a different IP on each request. Plug them into Playwright at the context level:
PROXY = {
"server": "http://residential.proxyon.io:8080",
"username": "YOUR_USERNAME",
"password": "YOUR_PASSWORD"
\}
with sync_playwright() as p:
browser = p.chromium.launch()
context = browser.new_context(proxy=PROXY)
page = context.new_page()
page.goto("https://example.com")$Also Read: How to Scrape Amazon Product Data
Final Thoughts
Requests and BeautifulSoup cover static sites. For JavaScript-heavy or protected targets, Playwright, paired with rotating residential proxies, handles the rest. Either way, Pandas gets the data into Excel in two lines. Residential proxies start at $1.75/GB with no subscription required. Deposit $5 and start scraping at Proxyon.
