HtmlUnit

💗 Sponsoring 💗

Constantly updating and maintaining the HtmlUnit code base already takes a lot of time.
I would like to make 2 major extensions in the next few months

For doing this I need your Sponsoring.

HtmlUnit is a "GUI-Less browser for Java programs". It models HTML documents and provides an API that allows you to invoke pages, fill out forms, click links, etc... just like you do in your "normal" browser.

It has fairly good JavaScript support (which is constantly improving) and is able to work even with quite complex AJAX libraries, simulating Chrome, Firefox or Edge depending on the configuration used.

It is typically used for testing purposes or to retrieve information from websites.

HtmlUnit is not a generic unit testing framework. It is specifically a way to simulate a browser for testing purposes and is intended to be used within another testing framework such as JUnit or TestNG. Refer to the document "Getting Started with HtmlUnit" for an introduction.

HtmlUnit is used as the underlying "browser" by different Open Source tools like WebDriver, Arquillian Drone, Serenity BDD, WETATOR, jenkins-test-harness, Selenium Foundation, Spring Testing, ... JWebUnit, JSFUnit,

HtmlUnit was originally written by Mike Bowler of Gargoyle Software and is released under the Apache 2 license. Since then, it has received many contributions from other developers, and would not be where it is today without their assistance.

Where to find...

Latest release November 29, 2024: version 4.7.0
Source code
News
Latest build / CI server: Jenkins build server

Documentation

Tutorials

Getting Started
WebClient - the browser
Tables
Logging
Windows and Frames
Keyboard navigation
The Java Web Scraping Handbook

A nice tutorial about webscraping with a lot of background information and details about HtmlUnit.

Features

Support for the HTTP and HTTPS protocols
Support for cookies
Ability to specify whether failing responses from the server should throw exceptions or should be returned as pages of the appropriate type (based on content type)
Support for submit methods POST and GET (as well as HEAD, DELETE, ...)
Ability to customize the request headers being sent to the server
Support for HTML responses
- Wrapper for HTML pages that provides easy access to all information contained inside them
- Support for submitting forms
- Support for clicking links
- Support for walking the DOM model of the HTML document
Proxy server support
Support for basic and NTLM authentication
Excellent JavaScript support (see the JavaScript section below)

Installation

Place all the required jars in your classpath. All of these can be found in the lib directory of the HtmlUnit installation (htmlunit-4.xx.x-bin.zip contains everything you need including all required dependencies).

For maven, you would add:

<dependency>
    <groupId>org.htmlunit</groupId>
    <artifactId>htmlunit</artifactId>
    <version>4.7.0</version>
</dependency>

If you like to use the latest available snapshot build you have to add the sonatype snapshot repository to your pom repositories section; also (check Twitter for regular information about new snapshot builds).

<repository>
  <id>OSS Sonatype snapshots</id>
  <url>https://oss.sonatype.org/content/repositories/snapshots/</url>
  <snapshots>
    <enabled>true</enabled>
    <updatePolicy>always</updatePolicy>
  </snapshots>
  <releases>
    <enabled>false</enabled>
  </releases>
</repository>

JavaScript Support

HtmlUnit provides excellent JavaScript support, simulating the behavior of the configured browser (Chrone, Firefox or Edge). It uses the Rhino JavaScript engine for the core language (plus workarounds for some Rhino bugs) and provides the implementation for the objects specific to execution in a browser.

The unit tests of some well-known JavaScript libraries are included in HtmlUnit's own unit tests; based on these unit tests, the following libraries are known to work well with HtmlUnit:

htmx 1.7.0: Full support (see unit test here)
htmx 1.8.4: Full support (see unit test here)
htmx 1.9.x: Full support (see unit test here)
jQuery 1.11.3: Full support (see unit test here)
jQuery 1.8.2: Full support (see unit test here)
MochiKit 1.4.1: Full support (see unit tests here)
MochiKit 1.4.2: Full support (see unit tests here)
GWT 2.5.0: Full support (see unit test here)
Sarissa 0.9.9.3: Full support (see unit test here)
Sarissa 0.9.9.7: Full support (see unit test here)
MooTools 1.2.1: Full support (see unit test here)
Prototype 1.7.1 (1.6.0, 1.6.1): Full support (see unit test here)
Ext JS 2.2: Very good support (see unit test here)
Dojo 1.9.3: Very good support (see unit test here)
Dojo 1.0.2: Good support (see unit test here)
YUI 2.3.0: Good support (see unit test here)

JavaScript code is executed just like in normal browsers when the page loads or when a handler is triggered. Additionally, HtmlUnit provides the ability to inject code into an existing page via HtmlPage.executeJavascript(String yourJsCode).

Refer to the Use JavaScript for more details.

Refer to the changes document for details on what is being added.