0

I am writing a web scrapping application using core Java. However when the application tried to access some https enabled URLs I am getting error. Below are the details.

Code:

           url = new URL("https://stackoverflow.com");
           is = url.openStream();   

Error:

javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)

When I checked on the web it was suggested that one of the fix is to disable the SSL verification as below.

            SSLContext ctx = null;
            TrustManager[] trustAllCerts = new X509TrustManager[]{new X509TrustManager(){
                public X509Certificate[] getAcceptedIssuers(){return null;}
                public void checkClientTrusted(X509Certificate[] certs, String authType){}
                public void checkServerTrusted(X509Certificate[] certs, String authType){}
            }};
            try {
                ctx = SSLContext.getInstance("SSL");
                ctx.init(null, trustAllCerts, new SecureRandom());
            } catch (NoSuchAlgorithmException | KeyManagementException e) {
                e.printStackTrace();
            }

            SSLContext.setDefault(ctx);

After adding is this, the error is resolved and code is able to access any https url. However it was mentioned that this is prone to security issues or man in the middle attack.

I want to know if there is an alternate solution which does not require disabling SSL verification or more secure while the code is able to access any https URL.

Thanks for your help.

  • stackoverflow.com uses LetsEncrypt with the DST root (still, although LE now plans to switch in Sept.) which is in every Oracle Java since 8u101 in 2016 (as 'identrustdstx3'); OpenJDK may differ depending on the build(er), as may other Javas like IBM and Android. Unless of course your network is using an interceptor (like many businesses or organizations, schools, ISPs, and countries) and you aren't getting the real stackoverflow cert (or site) at all. You can look at what you're getting with e.g. keytool -printcert -sslserver stackoverflow.com – dave_thompson_085 Jul 27 at 0:05
  • PS: scrapping != scraping. And while much of the web is worthless, harmful garbage that should be scrapped, you don't have the ability to do it. – dave_thompson_085 Jul 27 at 0:08
2

Read The Java Developer's guide to SSL Certificates

Your browsers keep a constantly updated list of root certificates from trusted certificate authorities.

Your Java installation also has such a list (referred to as the truststore) but it isn't updated as often (your java truststore will be updated when Java is updated). If you want the new list faster than that, you need to import the root certificates you want to trust.

List the CA root certificates trusted by your Java truststore to see which ones are missing:

keytool -list -keystore $JAVA_HOME/lib/security/cacerts

(if asked for a password, the default password is "changeit")

to import a missing one, following the directions from the article:

Adding a certificate to the truststore is necessary if we want to trust a certificate issued from a CA not present in the bundled truststore.

keytool -import -trustcacerts -file [certificate] -alias [alias] -keystore $JAVA_HOME/lib/security/cacerts

the root certificate file is first downloaded from the site you wish to scrape.

| improve this answer | |
0

Generally it is not recommended to disable certificate validation since without proper certificate validation a malicious man in the middle could sniff and modify the traffic in transit. But, depending on the actual (but unknown) use case of this web scraping this might be a tolerable risk here.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.