Protected Client Server communication advice

Question

I am hoping for a better understanding about SSL/TLS by setting up two devices, running two different applications, allowing communications to go over SSL/TLS.

With OpenSSL I have created a certificate and key which I installed on both, the server and the client. On the client side, I can also choose from three cipher suites (on the server I don't have that option to choose from).

• TLS_RSA_WITH_NULL_SHA
• TLS_RSA_WITH_3DES_EDE_CBC_SHA
• TLS_RSA_WITH_AES_128_CBC_SHA

I use Wireshark to see what is going on during the communication that always fails, no matter what I select from above. I see a normal TCP communication: SYN, SYN ACK, ACK and then a FIN from the client. It is very obvious, I am doing something wrong, but honestly no clue what to do. It seems that it doesn't even get to the point of using the SSL protocol.

Can anyone please clarify things for me what Iam doing wrong and hand me a possible solution? This requires more professional security skills. Thanks in advance ! John

P.S.: I want to use self-signed certificates since it is only used for testing.

Answer 1

If you see only the ACK, SYN+ACK, ACK and FIN packets, then you are seeing the shell of a TCP connection, but no data. If the FIN is from the client, then the client apparently decides that it has nothing to say. So this points at a configuration issue on the client: when the client should begin the SSL/TLS conversation (by sending the ClientHello message), it just gives up. In any case, at that point, the role of the server is to wait for the ClientHello, so its own SSL-level configuration has not been exercised yet.

---

Now some general comments:

• In SSL/TLS, the server should have a private/public key pair, the public key being wrapped in a certificate. The client may also have a private/public key pair -- but not the same one ! The whole point of asymmetric cryptography is to avoid having to share private elements between different entities. In fact, you should strive never to let private keys travel between systems: the more a key journeys, the more exposed it gets, thus the less private it becomes.

• Cipher suites select the set of cryptographic algorithms which will be used. In the three suites you talk, RSA will be used for the initial key exchange (so the server's public key should be of type RSA), and SHA-1 for integrity protection. The symmetric encryption will use, respectively, nothing at all, 3DES, or AES. If unsure, use AES. If you are thinking about using the "NULL" encryption, then think again.

• If you want to use SSL/TLS, and, even more, if you want to debug some SSL-related issues and have begun launching Wireshark (that's a good thing), then you will need to understand how SSL works. You could do worse than reading this answer for some explanations on that.