Java Web Scraping Tutorial (2026)

TL;DR

Java is a strong choice for web scraping when you need type safety, mature concurrency, and the ability to run scrapers within an existing JVM stack. Jsoup handles static pages, Playwright for Java handles JavaScript-heavy sites, and proxies keep both from getting blocked. The one thing most tutorials skip: Java disables Basic auth for HTTPS proxy tunneling by default, and you have to clear a system property to make authenticated proxies work.

Java web scraping means writing a program that fetches a web page, parses its HTML, and pulls out the data you need. Java is one of the most reliable languages for scrapers running in production pipelines. In this article, we'll explore how to scrape static and dynamic pages in Java, and how to route your scraper through proxies without hitting the authentication trap most guides never mention.

Setting Up a Java Scraper With Jsoup

Jsoup is the standard library for static pages in Java. It fetches HTML over HTTP and gives you CSS selectors to extract data. Java 25 is the current LTS and the version worth targeting in 2026.

Add Jsoup to your Maven project:

<dependency>
    <groupId>org.jsoup</groupId>
    <artifactId>jsoup</artifactId>
    <version>1.18.3</version>
</dependency>

Then fetch and parse a page:

Document doc = Jsoup.connect("https://example.com/products")
    .userAgent("Mozilla/5.0 (Windows NT 10.0; Win64; x64)")
    .timeout(10000)
    .get();

Elements products = doc.select("li.product");
for (Element product : products) {
    String name = product.selectFirst("h2").text();
    String price = product.selectFirst("span.price").text();
    System.out.println(name + " - " + price);
}

Always set a User-Agent. Jsoup's default identifies itself as a Java client, and many servers reject it on sight. The full selector syntax lives at jsoup.org.

Also Read: How to Set Up Rotating Proxies for Web Scraping (2026)

Scraping JavaScript-Heavy Sites

Jsoup only sees the raw HTML the server returns. If the page renders its content with JavaScript, you need a real browser. Older guides push HtmlUnit here, but its JavaScript engine chokes on modern React and Next.js sites. Playwright for Java is the better call. It drives a real Chromium instance and hands you the rendered DOM:

try (Playwright playwright = Playwright.create()) {
    Browser browser = playwright.chromium().launch();
    Page page = browser.newPage();
    page.navigate("https://example.com/products");
    String html = page.content();
    Document doc = Jsoup.parse(html);
}

The downside is resource cost. A headless browser eats far more memory than a plain HTTP client, so reserve it for pages that need rendering.

Adding Proxies to a Java Scraper

Scrape any site at volume from one IP and you will get blocked. Routing requests through residential proxies spreads your traffic across real household IPs, while datacenter proxies are the cheaper option for less protected targets.

Jsoup accepts a proxy directly:

Document doc = Jsoup.connect("https://example.com")
    .proxy("gate.proxyon.io", 8000)
    .get();

Here is the part nearly every Java web scraping tutorial skips. Since Java 8u111, the JVM disables Basic authentication for HTTPS tunneling by default. Your credentials never reach the proxy, and every HTTPS request fails with a 407 error. The fix is clearing one system property before setting your Authenticator:

System.setProperty("jdk.http.auth.tunneling.disabledSchemes", "");

Authenticator.setDefault(new Authenticator() {
    protected PasswordAuthentication getPasswordAuthentication() {
        return new PasswordAuthentication("user", "pass".toCharArray());
    }
});

This behavior is documented in Oracle's networking properties reference. One thing worth knowing: IP whitelisting sidesteps the whole problem, since the proxy authenticates your server's IP instead.

Also Read: Plug Proxyon Into Selenium

FAQ Section

Is Java good for web scraping?

Yes. Java offers strong typing, mature concurrency through virtual threads, and libraries like Jsoup and Playwright. It shines when scrapers run inside existing JVM infrastructure.

What is the best Java web scraping library?

Jsoup for static HTML, Playwright for Java when pages render content with JavaScript. Many production scrapers combine both.

Why do my proxy requests fail with 407 in Java?

The JVM disables Basic auth for HTTPS tunneling by default. Clear the jdk.http.auth.tunneling.disabledSchemes system property, or use IP whitelisting instead of credentials.

Should I use HtmlUnit for dynamic pages?

No. HtmlUnit's JavaScript engine fails on modern frameworks like React and Next.js. Playwright for Java renders these pages correctly.

Java or Python for web scraping?

Python has more scraping tooling and faster prototyping. Java wins on type safety, thread management, and integration with JVM data pipelines. Pick whichever matches your stack.

How do I avoid getting blocked while scraping in Java?

Set a real User-Agent, pace your requests, and rotate IPs through residential proxies. Rotation matters most, since repeated requests from one IP get banned fastest.

Can Jsoup execute JavaScript?

No. Jsoup only parses the HTML the server returns. For JavaScript-rendered content, render the page with Playwright first, then pass the HTML to Jsoup.

Final Thoughts

Jsoup covers static pages, Playwright covers rendered ones, and the tunneling property fix keeps authenticated proxies from silently failing. Get those three pieces right and a Java scraper will run unattended for months.