TL;DR
Indeed builds its search results with JavaScript and defends them with CAPTCHAs and per-IP rate limits, so a plain Python request returns an empty shell or a 403. The reliable method is to render the page through a headless browser or scraping API, then parse the job data from the embedded JSON blob instead of fragile CSS selectors. Route your requests through rotating residential proxies, pace them with short delays, and handle pagination with the start parameter. Stick to public job listing data only.
Scraping Indeed jobs gives you direct access to job titles, companies, locations, and salary data at a scale no manual search can match. Indeed hosts millions of postings, which makes it one of the most valuable sources for labor market research, salary analysis, and recruitment intelligence. The catch is that Indeed is also one of the most aggressively protected job boards on the web. In this article, we'll explore why most Indeed scrapers fail, where the job data actually lives inside the page, and how to collect it without getting blocked.
Why Plain Requests Fail on Indeed
Most tutorials still teach you to fetch an Indeed search URL with Python requests and parse it with BeautifulSoup. That approach is dead. Indeed renders its results with JavaScript, so the initial HTML is a near-empty shell, and the CSS class names those guides target changed long ago.
The second problem is detection. Indeed runs content scraping defenses that fingerprint your client and rate limit your IP. Send a few bare requests from one address and you get a CAPTCHA page or a 403 instead of job listings.
A working scraper needs two things at once: a client that renders JavaScript and an IP that looks like a real visitor. Rotating residential proxies solve the second part because every request exits through a different real-user IP.
Also Read: What is a Web Scraping Proxy and Why You Need One (2026)
Where the Job Data Actually Lives
Skip CSS selectors entirely. Every Indeed search page embeds its full listing data as a JSON object inside a script tag, assigned to the mosaic-provider-jobcards variable. One regular expression pulls it out, and json.loads turns it into structured Python data with the title, company, location, salary, and job key already separated.
import re, json
data = re.findall(r'window.mosaic.providerData\["mosaic-provider-jobcards"\]=(\{.+?\});', html)
jobs = json.loads(data[0])["metaData"]["mosaicProviderJobCardsModel"]["results"]This matters more than it sounds. Class names on Indeed are build-generated and change with every redesign, but the embedded JSON structure stays stable for much longer. The jobkey field also lets you build a direct link to any posting: https://www.indeed.com/viewjob?jk=JOBKEY.
Handling Pagination
Indeed paginates with a start query parameter that moves in steps of 10. The first page is start=0, the second is start=10, and so on. Loop through the offsets, fetch each page, and extract the same JSON blob every time. Add a delay of a few seconds between pages. Firing requests back to back is the fastest way to get throttled, even with proxies in place.
Avoiding Blocks at Scale
Residential IPs are the right choice for Indeed because the target checks IP reputation hard. Datacenter proxies are faster and cheaper, but their IP ranges are easier to flag on protected targets like this one. The trade-off comes down to what you are actually trying to do: use datacenter IPs for lighter targets and keep residential rotation for Indeed itself.
One thing worth knowing: only collect public listing data. Job titles, companies, locations, and posted salaries are visible to anyone. Applicant data, resumes, and anything behind a login are off limits and covered by privacy laws.
Also Read: How to Set Up Rotating Proxies for Web Scraping (2026)
FAQ Section
Is it legal to scrape Indeed jobs?
Public job listing data is generally the defensible scope, but Indeed's terms restrict automated access. Review the terms and your local laws, and never collect applicant or personal data.
Why does my Python requests scraper return an empty page?
Indeed renders its listings with JavaScript. A bare HTTP client only receives the initial shell, so the job data never appears in the response body.
Do I need a headless browser to scrape Indeed?
Yes, or a scraping API that renders the page for you. The embedded JSON only exists in the HTML after the page's scripts run.
What proxy type works best for scraping Indeed?
Rotating residential proxies. Indeed flags datacenter IP ranges quickly, while residential IPs blend in with normal visitor traffic.
How do I scrape more than one page of results?
Increment the start query parameter in steps of 10. Read the total match count from the first page and cap your loop at a sensible maximum.
Why should I parse embedded JSON instead of CSS selectors?
Indeed's class names are build-generated and break on every redesign. The mosaic-provider-jobcards JSON gives you every field already structured and survives markup changes.
Does Indeed offer an official API?
Indeed runs a Publisher API and partner program for licensed access. It is the right path for large commercial projects that need more than public search pages.
Final Thoughts
Scraping Indeed comes down to three things: render the page, parse the embedded JSON, and rotate clean residential IPs with sensible pacing. Get those right and the rest is a small Python script. Skip any of them and you will spend more time fighting CAPTCHAs than collecting data.
