I’m working on windows desktop application which is created using C++ (IDE : Qt creator). It has login panel, where i do user validation via https connection using openssl 1.0 library. Application is working in most of the machines, but i'm also experiencing "SSL Handshake failed" error while making https connection from few machines.
I tried debugging the error using wireshark. My observation is as follows:
1) Client sends [SYN] to server.
2) Server sends [SYN,ACK] to client.
3) Client sends [ACK] to server.
4) Client sends the message “Client Hello” to the server.
5) Server sends its public key with the message “Server Hello, Certificate, Server Hello Done”
6) Client sends its public key with the message “Client Key Exchange, Change Cipher Spec, Encrypted Handshake Message”
7) Server sends encrypted handshake message with the message “Change Cipher Spec, Encrypted Handshake Message”
8) Client sends [FIN,ACK]
9) Server sends [FIN,ACK]
10) Client sends [FIN]
In 7th step, as soon as client receives encrypted message from the server, client initiates termination of handshake by FIN signal. Any idea, why does client tear down ssl connection with “SSL handshake failure” after both parties exchanging the keys?