File downloads in real browsers

Because file downloading can be complex, let's start with some background info about file downloads in real browsers.

When surfing the internet, you usually open URLs pointing to (X)HTML content. Your web browser parses this data and renders the content on your screen. There are also additional file types your browser handles natively, such as CSS, JavaScript, images, sound, and video files. This processing occurs automatically behind the scenes.

There are only three primary cases where a browser delegates control of content handling to the user:

  • The file type (MIME type) is not supported natively by your browser (e.g., an Excel or Word file).
  • The Content-Disposition Header of the HTTP response explicitly marks the content as an attachment.
  • The request is initiated from the client side by clicking an anchor element with the download attribute set, prompting local file storage.

In all three cases, the content is saved as a file on your local disk. Depending on user settings, this occurs automatically or by displaying a file save dialog.

File downloads with HtmlUnit

Due to HtmlUnit's nature as a headless browser, there is no GUI rendering. However, like real browsers, HtmlUnit parses pages delivered by the server. If the response type is supported, the content is made available as an HtmlPage (or XHtmlPage), providing full programmatic access to embedded elements like JavaScript, CSS, and images. HtmlUnit also provides support for text-only content (TextPage) and plain XML content (XmlPage).

Because HtmlUnit is primarily designed for automated testing and scraping, it cannot present a graphical file dialog. The current implementation offers two primary mechanisms to handle file downloads:

UnexpectedPage (default)

HtmlUnit handles unknown or unsupported content types similarly to known content—the data stream is wrapped inside a page object and placed within a window. For unhandled file types, HtmlUnit creates an UnexpectedPage instance. In most cases, UnexpectedPage replaces the current page in the active window, though in certain cases (such as clicking an anchor with a download attribute), a new target window is created.

You can access the raw downloaded content stream directly from the enclosed UnexpectedPage:

try (final WebClient webClient = new WebClient(BrowserVersion.FIREFOX)) {
    HtmlPage page = webClient.getPage(uri);
    WebWindow window = page.getEnclosingWindow();

    // click an anchor/button that triggers a file download

    UnexpectedPage downloadPage = (UnexpectedPage) window.getEnclosedPage();

    try (InputStream downloadedContent = downloadPage.getInputStream()) {
        // e.g., save input stream to a local file
    }
}

In cases where the download opens inside a new window, you can retrieve the window instance like this:

WebWindow newWindow = webClient.getWebWindows().get(webClient.getWebWindows().size() - 1);

AttachmentHandler

To customize default download handling, you can register a custom implementation of the AttachmentHandler interface on the WebClient instance.

The AttachmentHandler interface works alongside default UnexpectedPage handling. If your custom AttachmentHandler declines a response, HtmlUnit falls back to the default behavior and assigns an UnexpectedPage to the window. You can override the isAttachment() method in your handler to target specific responses. By default, isAttachment() detects responses with a Content-Disposition header of type attachment, or responses without a Content-Disposition header that have a Content-Type of application/octet-stream.

The handleAttachment(WebResponse, String) method is triggered when a response is identified as an attachment:

boolean handleAttachment(final WebResponse response, final String attachmentFilename)

You can process the attachment in your implementation (for example, saving it directly to a file system) and return a boolean indicating whether the event was handled. Depending on the return value, the handler operates in two modes:

  1. true: Signals that the response has been fully handled. The current page will NOT be replaced by an UnexpectedPage, and handleAttachment(Page, String) will not be called.
  2. false: Delegates further processing back to HtmlUnit:
    • A new web window is created.
    • An UnexpectedPage is constructed.
    • The method handleAttachment(Page, String) on your AttachmentHandleris invoked.
    • The UnexpectedPage is placed inside the new window.

The following example collects attachment responses into a list without replacing the active window content:

final List<WebResponse> attachments = new ArrayList<>();

try (final WebClient webClient = new WebClient(BrowserVersion.FIREFOX)) {

    webClient.setAttachmentHandler(new AttachmentHandler() {
        @Override
        public boolean handleAttachment(final WebResponse response, final String attachmentFilename) {
            attachments.add(response);
            return true;
        }

        @Override
        public void handleAttachment(final Page page, final String attachmentFilename) {
            throw new IllegalAccessError("handleAttachment(Page, String) called unexpectedly");
        }
    });

    // start browsing
    HtmlPage page = webClient.getPage(uri);
}

For additional details, refer to the AttachmentHandler JavaDoc API documentation.

Note on special cases: Clicking an anchor tag with the download attribute set bypasses isAttachment() checks and directly forwards the response to the registered AttachmentHandler.