Using the options to adjust the browser

There are various options available to make fine-grained adjustments to the browser.

@Test
public void homePage_Firefox() throws Exception {
    try (final WebClient webClient = new WebClient(BrowserVersion.FIREFOX)) {
         // disable javascript
         webClient.getOptions().setJavaScriptEnabled(false);
         // disable css support
         webClient.getOptions().setCssEnabled(false);

        final HtmlPage page = webClient.getPage("https://www.htmlunit.org/");
        Assert.assertEquals("HtmlUnit – Welcome to HtmlUnit", page.getTitleText());
    }
}

The default values for most options are similar to the default values of real browsers, but (as always) there is one important exception:

HtmlUnit stops JavaScript execution at the first unhandled exception—browsers do not stop. You can change this by changing the throwExceptionOnScriptError option to false.

@Test
public void homePage_Firefox() throws Exception {
    try (final WebClient webClient = new WebClient(BrowserVersion.FIREFOX)) {
         // proceed with the js execution on unhandled js errors
         webClient.getOptions().setThrowExceptionOnScriptError(false);

        final HtmlPage page = webClient.getPage("https://www.htmlunit.org/");
        Assert.assertEquals("HtmlUnit – Welcome to HtmlUnit", page.getTitleText());
    }
}

For a complete list and more details please have a look at the WebClientOptions API.

Change the browser language / time zone

Changing the language/time zone cannot be done from the options; it must be done before the WebClient is created.

All BrowserVersions are shipped with 'en-US' as the language and 'America/New_York' as the timezone.

To change these default settings, a customized copy of the corresponding BrowserVersion must be created using the BrowserVersionBuilder. This new BrowserVersion can then be used to create a WebClient.

final BrowserVersion.BrowserVersionBuilder builder = new BrowserVersion.BrowserVersionBuilder(BrowserVersion.FIREFOX);

builder.setSystemTimezone(TimeZone.getTimeZone("Europe/Berlin"));
builder.setBrowserLanguage("de-DE");
builder.setAcceptLanguageHeader("de-DE,de");

final BrowserVersion germanFirefox = builder.build();
try (final WebClient webClient = new WebClient(germanFirefox)) {
    // ...
}

There is no support for changing the language/timezone after the WebClient has been created.

For more details please have a look at the BrowserVersion.BrowserVersionBuilder API.

Change the browser user agent

Changing the user agent is similar to changing language/time zone (see above).

You have to create a customized copy of the corresponding BrowserVersion using the BrowserVersionBuilder. This adapted BrowserVersion can then be used to create a WebClient.

final BrowserVersion.BrowserVersionBuilder builder = new BrowserVersion.BrowserVersionBuilder(BrowserVersion.FIREFOX);

builder.setUserAgent("Mozilla/5.0 (iPhone; CPU iPhone OS 14_5 like Mac OS X) "
        + "AppleWebKit/605.1.15 (KHTML, like Gecko) FxiOS/128.0 Mobile/15E148 Safari/605.1.15");

final BrowserVersion iosFirefox = builder.build();
try (WebClient webClient = new WebClient(iosFirefox)) {
    // ...
}

For more details please have a look at the BrowserVersion.BrowserVersionBuilder API.

Using a http proxy

There is a special WebClient constructor that allows you to specify proxy server information in those cases where you need to connect through one.

@Test
public void homePage_proxy() throws Exception {
    try (final WebClient webClient = new WebClient(BrowserVersion.FIREFOX, PROXY_HOST, PROXY_PORT)) {

        //set proxy username and password 
        final DefaultCredentialsProvider credentialsProvider = (DefaultCredentialsProvider) webClient.getCredentialsProvider();
        credentialsProvider.addCredentials("username", "password");

        final HtmlPage page = webClient.getPage("https://www.htmlunit.org");
        Assert.assertEquals("HtmlUnit – Welcome to HtmlUnit", page.getTitleText());
    }
}

In case the proxy server requires credentials you can define them on the DefaultCredentialsProvider from the webClient.

@Test
public void homePage_proxy() throws Exception {
    try (final WebClient webClient = new WebClient(BrowserVersion.FIREFOX, PROXY_HOST, PROXY_PORT)) {

        //set proxy username and password 
        final DefaultCredentialsProvider credentialsProvider = (DefaultCredentialsProvider) webClient.getCredentialsProvider();
        credentialsProvider.addCredentials("username", "password", PROXY_HOST, PROXY_PORT);

        final HtmlPage page = webClient.getPage("https://www.htmlunit.org");
        Assert.assertEquals("HtmlUnit – Welcome to HtmlUnit", page.getTitleText());
    }
}

Socks proxy sample

The setup of SOCKS proxies is a bit more tricky but in general follows the same pattern.

@Test
public void homePage_proxy() throws Exception {
    try (final WebClient webClient = new WebClient(BrowserVersion.FIREFOX)) {
        // socks proxy / the true as last parameter marks this as socks proxy
        webClient.getOptions().setProxyConfig(new ProxyConfig(SOCKS_PROXY_HOST, SOCKS_PROXY_PORT, null, true));

        //set proxy username and password if required
        final DefaultCredentialsProvider credentialsProvider = (DefaultCredentialsProvider) webClient.getCredentialsProvider();
        credentialsProvider.addSocksCredentials("username", "password", SOCKS_PROXY_HOST, SOCKS_PROXY_PORT);

        final HtmlPage page = webClient.getPage("https://www.htmlunit.org");
        Assert.assertEquals("HtmlUnit – Welcome to HtmlUnit", page.getTitleText());
    }
}

AlertHandler

The handler to be used to process JavaScript alerts triggered when the JavaScript method Window.alert() is called.

ConfirmHandler

The handler for the JavaScript function window.confirm().

PromptHandler

The handler for the JavaScript function window.prompt().

StatusHandler

A handler for changes to window.status.

AttachmentHandler

A handler for attachments, which represent pages received from the server containing Content-Disposition=attachment headers.

ClipboardHandler

A handler for clipboard access.

PrintHandler

A handler for providing Window.print() implementations.

WebStartHandler

A handler for WebStart support.

FrameContentHandler

A handler to make a decision to load the frame content or not. See Blocking loading of frame content for a sample.

CSSErrorHandler

A handler for CSS parser error processing.

OnbeforeunloadHandler

A handler for JavaScript onbeforeunload event processing.

RefreshHandler

A handler for page refreshes.

Fetch API Polyfill

window.fetch polyfill

webClient.getOptions().setFetchPolyfillEnabled(true);