I am trying to use the Selenium HTMLUnitDriver to read the contents of a page (https://mattermost.com/security-updates/). If you scroll down right as the page loads in a browser, you’ll see that the table at the bottom half of the page loads in later than the rest of the page. When using the HTMLUnitDriver to navigate to the page, the web elements within the table will never load.

// Set up the driver.
HtmlUnitDriver driver = new HtmlUnitDriver();
driver.getWebClient().getOptions().setMaxInMemory(-1);
driver.getWebClient().getOptions().setHistoryPageCacheLimit(0);
driver.setJavascriptEnabled(true);
driver.getWebClient().getOptions().setThrowExceptionOnScriptError(false);

// Go to webpage.
driver.get("https://mattermost.com/security-updates/");

// Attempt to get elements. This will return no elements.
driver.findElements(By.xpath("//table[@id='mattermostServerTable']/tbody/tr"));

I’ve tried using things like waitForBackgroundJavaScript (https://htmlunit.sourceforge.io/apidocs/org/htmlunit/WebClient.html#waitForBackgroundJavaScript-long-) as well as having the driver scroll and click on the page to see if it triggers the table to load. None of these have worked.

I was able to get this to work using Chrome driver. However, I would still prefer to use the HtmlUnitDriver.