Unable to connect using Htmlunit to website using HTTPS

Question

I am trying to use HtmlUnit to log in to a website that uses a HTTPS connection and I get an error caused by the statement

page = webClient.getPage(url);

The exception is

error:javax.net.ssl.SSLHandshakeException:sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target`

What I have done so far:

Exported the individual certificates in the website's certificate chain using Firefox and imported them into the JVM's default keystore using the java keytool, for example:

keytool -import -trustcacerts -alias root -file C:\Users\myUser\Desktop\Certificates\VeriSignClass3PublicPrimaryCertificationAuthority-G5.crt -keystore C:\Program Files (x86)\Java\jre1.8.0_91\lib\security\cacerts

Started a new keystore using the java keytool, and imported only the relevant root, intermediate, and specific website's certificate.

Set this certificate as the java certificate store using

 System.setProperty("javax.net.ssl.trustStore", "C:/Users/Henry/workspace/AutoEtime/cacerts");

Tried changing the browser version to Chrome and then to Firefox
Tried to use SSLPoke.class from confluence.atlassian.com to test if the certificates were correct, however I don't know the port to connect to, and using the default 433 gives a ConnectException: Connection timed out

I have not tried using any of the other methods outlined in the WebClientOptions documentation found here, because I am not sure how to use them to get the effect I want.

One of the most common answers that I have seen online is the suggestion to use webClient.setUseInsecureSSL(true);

This works fine for testing my application, however I need to have the connection be secured using SSL because of the sensitivity of the username and passwords that I am using to connect to the website.

One interesting thing I want to add is that one out of every 5-10 attempts actually works, and does not throw the SSLHandshakeException, even though I have changed nothing in the code.

How do I fix this problem? Additionally, why does it work every once in a while?

The default port for HTTPS is 443 not 433. Manually changing/writing files on Windows under \Program Files or \Program Files (x86) is a bad idea; for newish Windows versions such changes may not work or may work only sometimes, which might explain your intermittent behavior. Either install Java elsewhere (I like c:\Java) or use your second bullet: create your own keystore and point to it; you should only need the root cert(s) if the server behaves correctly, and make sure to set trustStore (AND trustStorePassword if not changeit) before the FIRST use of SSL in the JVM process. — dave_thompson_085
– dave_thompson_085, Commented Feb 23, 2017 at 7:07
Is it possible that you provide the hostname you try against? — Ahmed Ashour
– Ahmed Ashour, Commented Feb 23, 2017 at 12:51
You were right Dave, I had typed the port in wrong in both my question and as an argument when running SSLPoke.class. As soon as I changed it and added the certs, it successfully connected. — Henry Mahar
– Henry Mahar, Commented Feb 23, 2017 at 23:38

Henry Mahar · Accepted Answer · 2017-02-23 23:37:29Z

1

Okay so I figured it out by just trying different things. It turns out that I needed to import the certificates in .der form instead of .crt. The default keystore already had the correct root certificate, but what it was missing was some combination of the intermediate and leaf certificate. After I imported intermediate.der and leaf.der and specified where my copy of the cacerts file was the SSLHandshakeException hasn't happened again. Hopefully this has actually fixed it and that exception is gone for good.

answered Feb 23, 2017 at 23:37

Henry Mahar

113 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

bytor99999 Over a year ago

Henry Mahar I don't understand your answer here. What is a der? How did you find them? We have a similar issue.

Collectives™ on Stack Overflow

Unable to connect using Htmlunit to website using HTTPS

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related