TL;DR
The most reliable way to scrape Google News is through its RSS feeds, not its HTML pages. Google generates its CSS class names dynamically, so scrapers built on HTML selectors break constantly. RSS gives you structured XML with titles, links, sources, and publish dates, and you can filter it by keyword, topic, or location. For anything beyond small runs, route your requests through rotating proxies to avoid rate limits and IP blocks.
Google News aggregates headlines from thousands of publishers, which makes it a top source for brand monitoring, market research, and sentiment analysis. The problem is that most scraping tutorials point you at the HTML pages, where Google rotates its class names and breaks your selectors within days. In this article, we'll explore how to scrape Google News using RSS feeds and Python, how to handle the redirect links Google gives you, and how to keep your scraper running at scale.
Why Scraping Google News HTML Is a Bad Idea
Inspect news.google.com, and you will see class names like IFHyqb and JtKRv. These are obfuscated and dynamically generated, so Google can rotate them at any time. A scraper built on those selectors works today and returns nothing next week.
The HTML route also pulls in Google's bot detection. Repeated requests from one IP get flagged fast, and you start hitting CAPTCHAs and 429 responses. The keyword here is stability, and Google News has a structured output that does not depend on markup at all.
The RSS Feed Method
Google News exposes RSS feeds that return clean XML with the title, link, publication date, and source for every article. No selectors, no JavaScript rendering. The base feed lives at news.google.com/rss, and you can narrow it down with a search endpoint that takes four parameters: q is your keyword, hl sets the language, gl sets the country, and ceid combines both. Swap those values to pull localized results from any region. Each feed returns up to 100 items following the standard RSS specification, so any XML parser can read them. Here is a working example with Python:
import requests
from bs4 import BeautifulSoup
url = "https://news.google.com/rss/search?q=bitcoin&hl=en-US&gl=US&ceid=US:en"
response = requests.get(url)
soup = BeautifulSoup(response.text, "xml")
for item in soup.find_all("item"):
print(item.title.text)
print(item.source.text)
print(item.pubDate.text)
print(item.link.text)
print("-" * 40)That is the entire scraper. BeautifulSoup parses the XML directly, and since the feed structure is standardized, this code survives every frontend redesign.
Also Read: How to Do Web Scraping Without Getting Blocked
Handling Google News Redirect Links
One thing worth knowing: the links in Google News feeds do not point to the publisher. They point to news.google.com/rss/articles/..., which is a redirect wrapper around the real URL. The fix is to send a GET request to the wrapped link and read response.url after the redirects resolve. That gives you the publisher's address, and from there you can fetch the full article body and parse it separately, since the feeds only carry headlines and snippets.
Resolving redirects multiplies your request count. A run that pulls 100 headlines turns into 200+ requests once you follow every link, and that volume is what gets single IPs flagged.
Scaling Up Without Getting Blocked
For a few dozen requests a day, your own IP is fine. Once you start polling multiple feeds every few minutes or resolving thousands of redirect links, Google will rate limit you, and the fix is to spread your traffic across a pool of IPs.
Residential proxies are the safest option because they route requests through real household connections that Google treats as normal users, and they let you target specific countries when pulling localized feeds. At moderate volume on the RSS endpoints, cheaper datacenter proxies usually get the job done, since RSS traffic draws less scrutiny than HTML scraping.
Pace your requests either way. Even with rotation, hammering the endpoints in bursts is a detection signal, and a short random delay keeps your success rate high.
Also Read: How to Set Up Rotating Proxies for Web Scraping
FAQ Section
Is scraping Google News legal?
Scraping publicly available data like headlines and links is generally considered legal in most jurisdictions, but republishing full article content can violate copyright. Collect metadata, link back to the source, and check the laws that apply to your use case.
Does Google News have an official API?
No. Google discontinued its official News API years ago. Your options today are the RSS feeds, scraping the HTML, or third-party services. RSS is the only free, structured, and officially served option.
How many results can I get from one RSS feed?
Each feed returns up to 100 items. If you need more coverage, split your topic into multiple narrower search queries and merge the results, deduplicating by URL.
Why do my scraped links not open the article directly?
Google wraps every article link in a redirect through its own domain. Follow the redirect with a GET request and read the final URL to get the publisher's page.
Do I need a headless browser to scrape Google News?
Not for the RSS method. The feeds are plain XML served without JavaScript, so requests and BeautifulSoup are enough. You only need Playwright or Selenium if you insist on scraping the rendered HTML pages, which is the fragile route anyway.
Final Thoughts
Scraping Google News comes down to picking the stable path. The RSS feeds give you structured data that survives every frontend redesign, while HTML scraping leaves you patching selectors and dodging CAPTCHAs. Start with the feeds, resolve the redirect links when you need full articles, and add rotating proxies once your volume grows.
