SSL Handshake failed

Question

I’m working on windows desktop application which is created using C++ (IDE : Qt creator). It has login panel, where i do user validation via https connection using openssl 1.0 library. Application is working in most of the machines, but i'm also experiencing "SSL Handshake failed" error while making https connection from few machines.

I tried debugging the error using wireshark. My observation is as follows:

1) Client sends [SYN] to server.
2) Server sends [SYN,ACK] to client.
3) Client sends [ACK] to server.
4) Client sends the message “Client Hello” to the server.
5) Server sends its public key with the message “Server Hello, Certificate, Server Hello Done”
6) Client sends its public key with the message “Client Key Exchange, Change Cipher Spec, Encrypted Handshake Message”
7) Server sends encrypted handshake message with the message “Change Cipher Spec, Encrypted Handshake Message”
8) Client sends [FIN,ACK]
9) Server sends [FIN,ACK]
10) Client sends [FIN]

In 7th step, as soon as client receives encrypted message from the server, client initiates termination of handshake by FIN signal. Any idea, why does client tear down ssl connection with “SSL handshake failure” after both parties exchanging the keys?

Answer 1

Your description of the handshake seems to indicate that the client and the server conducted the handshake completely, and then the client dropped the connection. This means that "something" was not right from the client's point of view. There are mostly two possible candidates:

1. The certificate sent by the server is not "proper"; the client decided that some user validation is necessary. The client completed the handshake so that it may reopen the SSL session with a faster "abbreviated handshake" (reusing the negotiated "master secret" without having to to the asymmetric crypto again), but closed the connection so as not to keep resources open on the server while the human user makes up his mind (the meat bag is slow).

2. The Finished message sent by the server (that's the "encrypted handshake message") contains an incorrect value (from the point of view of the client) due to some bug (probably in the client). This is not a very probable occurrence.

My guess is that you are in the first case: the server uses a certificate chain which is "not good" for the client. Usual culprits:

• The server certificate chain does not link up to one of the "trusted roots" of the client (depending on the library used on the client, the list of roots can be in several places).
• The server name, as expected by the client (the one in its URL) is not matched against the names in the server's certificate.
• The client clock is wildly off, so it rejects some certificate which is, from its point of view, either issued "in the future", or long expired.

Answer 2

export the certificate of the server to the client machine to a file such as servercert.crt. On the client run: certutil -verify -urlfetch servercert.crt

It will almost certainly tell you why the server certificate chain was not considered valid. Regards