RESTful web service and SSL certificate

Question

We have a server that provides RESTful services to our application server. It runs on Amazon Elastic Beanstalk and mapped with a domain name and SSL certificate registered at GoDaddy.

It ran perfectly fine in the last few years. In March, we renewed our SSL certificate and updated it for all the servers we have. A few days ago, we added a new resource to the RESTful service, which has a new request path, say: servername.com/rest/newResource.

All of a sudden, every single web service call in our application server threw the following exception:

java.net.ConnectException: General SSLEngine problem to https://servername.com/rest/newResource at com.ning.http.client.providers.netty.NettyConnectListener.operationComplete(NettyConnectListener.java:103)
        at org.jboss.netty.channel.DefaultChannelFuture.notifyListener(DefaultChannelFuture.java:427)
        at org.jboss.netty.channel.DefaultChannelFuture.notifyListeners(DefaultChannelFuture.java:413)
        at org.jboss.netty.channel.DefaultChannelFuture.setFailure(DefaultChannelFuture.java:380)
        at org.jboss.netty.handler.ssl.SslHandler.setHandshakeFailure(SslHandler.java:1429)
        at org.jboss.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1305)
        at org.jboss.netty.handler.ssl.SslHandler.decode(SslHandler.java:925)
        at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:425)
        at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:310)
        at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
        at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
        at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
        at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
        at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
        at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
        at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
        at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:722) Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:385)
        at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:292)
        at sun.security.validator.Validator.validate(Validator.java:260)
        at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:326)
        at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:283)
        at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:138)
        at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1325)
        ... 23 more Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:196)
        at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:268)
        at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:380)
        ... 29 more

All the other web service calls still worked fine, except anything under this new request path. I ended up resolving this issue by importing the latest GoDaddy root certificate and our (wildcard) domain certificate to the JDK's keystore (CAcerts).

Although the issue is now gone, my confusion on this topic remains. Could someone explain to me, why this was happening in the first place?

Answer 1

Your question is pretty vague and there are answers at many possible levels.

Why do the REST calls fail? Because you're using SSL (or TLS which at this level is the same thing) and SSL usually, including here, requires the server "authenticate" (prove its identity) with a certificate which the client verifies. Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target says the client isn't able to determine that the server's cert is trustworthy, and therefore doesn't trust it.

Why does SSL/TLS usually authenticate the server? Because otherwise in many cases it is possible (even easy) for a criminal or other adversary to get "in the middle" of your supposedly secure connections and steal and/or alter your supposedly secure data. Browsers usually give you a choice by displaying a warning that says something more or less like "this server is probably fradulent and may steal your money and identity, do you really really want to connect?" (and in practice many users will). An API like REST instead just fails, because there is no reliable and rapid way to check with a human to decide to abandon security. (If you know in advance you don't want security, just don't use SSL.)

What is a certificate path or chain? A series of certificates, starting from a CA root cert (or sometimes nonroot trust anchor), going through zero or more (but typically one) intermediate CA cert(s), ending at the desired 'target' cert, where each cert other than the first is digitally signed by* and untamperably linked to the previous one. The first cert (root or anchor) is one that you have explicitly decided to trust, or you have accepted someone else's judgement to trust such as a browser vendor or here the Java developers. Sometimes the path is expressed the other direction, for example when actually sent in the SSL/TLS handshake; but it's easy to just reverse when needed. * To be exact each signature uses the private key that matches the certificate, not the certificate itself, but these are closely linked mathematically and we usually elide this.

This mostly-hierarchical structure, called Public Key Infrastructure (PKI), allows hundreds of millions of Internet users and systems who don't know each other in advance to establish trust and communicate securely -- as long as the CAs do their job correctly, which is not always the case (and there are other questions about that if you are interested). If, as your question sounds like, you are in a situation where only your client(s) talk to only your server(s), you don't really need the full glory of PKI, and might consider creating your own CA and certs; OTOH the existing ones are already existing, easy enough to use and usually not too expensive, and that's one less thing you need to know, and remember, and document, and test, and audit, etc.

Note the chain is defined using a particular root cert, not just a CA. It is possible and not rare for one CA organization to have than one root cert (and very common to have numerous intermediate certs as well). In particular over the past several years many CAs that used to be satisified with RSA-1024 and SHA-1 or even MD5 roots have upgraded to RSA-2048 (or sometimes more, or sometimes ECC) and/or SHA-2; this requires a new root, although often (usually?) the CA will provide "bridge" or "transition" certs that chain new entity certs to the old root(s) until all verifiers get upgraded, which can take years.

How does Java SSL client verify any server? It uses a local "truststore" to contain the CA root certs or anchors that should be trusted. This truststore can be set explicitly for a program, but it defaults to the file jssecacerts if it exists and otherwise the file cacerts in the JRE/lib directory. This file is in the Java KeyStore (JKS) format, but it actually contains certificates and not keys. (In general a JKS can contain any combination of privateKey and trustedCert entries.) The cacerts initially included in an installation of a JRE (including the JRE portion of a JDK) as distributed by Oracle contains about 80 'well-known' CA certs that the Java developers have decided are trustworthy, but you can delete or add as desired for your particular installation.

In addition in most cases the Java client compares the desired server name (from the URL) to the server name(s), including a wildcard name (as in your case), encoded untamperably in the server cert. However, if that fails it gives a different error from the one you saw.

Why did this Java SSL client reject this server? Apparently this server used a certificate chain ending in a root that was not in your (default) truststore. As a result when the Java SSL client tried to build a valid "path" from the server cert to a trusted root or anchor, it did not find such a root or anchor in the truststore. You say the server (or at least some server?) is using a recently reissued cert; it's not unusual for new certs to use a relatively new root, possibly newer than the JRE you are running.

To determine this exactly you need to look carefully at (or show us or point us to) the server cert and the chain certs and root cert it uses. If this is your server you should already have them; in other cases if the server supports a request a browser can safely do (perhaps a toplevel GET /help or somesuch) any common browser can display the cert chain, and each cert in the chain; if that isn't possible you can (get and) use the s_client utility in OpenSSL to connect to any reachable server and dump its cert chain.

Why did adding the GoDaddy root fix the problem? You added a root cert which is apparently the one the server chain needs; now that it is present, the Java SSL client is able to chain the server cert to a trusted root and verification succeeds. It is not necessary to have the server cert in the client truststore, and could be confusing later; I suggest you remove it.