Need help with decrypting the “client key exchange” captured in Wireshark

Question

I am trying to understand how the SSL handshake happens for HTTPS.

I did the following:

1. Captured the packets for a HTTPS connection using Wireshark.
2. examined the Client Hello for cipher suites supported (not important)

3. Examined the Server Hello for the cipher suite decided by the server. It is TLS_RSA_WITH_RC4_128_MD5. Which means Handshake algorithm is RSA. which means the master key is encrypted using the public key of the server and sent to server.

4. The next message Client key exchange is where I am facing the challenge. Exported the "Handshake protocol: Client Key exchange" encrypted byte stream. The byte stream in hex is:

   10 00 01 00 8e 1b d9 49 6f 9e 15 8f b9 b6 8a 2e  
   e0 90 f1 54 3b 54 7d d3 d5 2e 64 b5 37 cc ae 74  
   ec 3f 38 59 1a 42 78 98 3f e1 4e 7b 8b 84 74 a9  
   17 95 c0 b7 07 d9 b1 a1 d0 1f 5a a1 2e 71 b6 98  
   ea 4b 6c 62 f3 b3 8c 8e d7 20 9a 4b 6a a7 d7 4c  
   f8 69 c9 6c d6 0b 8b d0 9f 59 28 f5 52 60 fa e9  
   72 52 4c 87 98 30 fe 6f ef a6 5b 11 fd 6b 0e 0d  
   db 60 d5 d4 d8 a6 0e 6d 9f 02 58 01 a4 21 d5 aa  
   17 80 5f 42 ec 84 78 a8 41 ed bc 94 c5 83 ab 74  
   09 b9 91 9d bf 6d c1 4b 85 95 90 d8 b4 22 fb 00  
   a4 76 af 54 e2 c3 1e 84 6f 5e 02 18 05 f5 6c 83  
   7f dc a7 44 85 24 06 b6 89 6f 13 4e 25 f0 ce 59  
   23 8c 50 4d c2 56 11 b9 0d 63 b5 28 b8 ad e7 9c  
   f2 16 96 f8 dd 4a f9 b5 72 8c 6f 6a 6c 8b 40 d7  
   03 c7 a8 d6 8e 88 38 00 d2 d3 9b 4a 04 3a 16 55  
   1f c9 58 c8 3f d3 7a 33 9a 3f 98 1c 74 83 3c 45  
   5a b2 9c da
   

   I stripped off the first four bytes (message type and size)

5. Tried decrypting the client key using the server's private key using the command:

   openssl rsautl -decrypt -inkey /etc/apache2/ssl/apache.key -in Clientkeyexchange_enc -out Clientkeyexchange_dec
   

   The hex value of the result is:

   03 00 E1 B9  2F 27 4F 85   46 AF 54 CC  5D 55 5E 92
   71 CD 14 60  02 96 08 BA   8D E0 65 B7  A5 27 EF E4
   F7 4E 4A 02  55 47 80 4E   36 FF 49 75  D2 B6 AB 83
   

I am unable to decrypt the application data using this value. I know I have misunderstood something. But unable to find out what.

Answer 1

First, before actually answering, I hope you do know that Wireshark can do all the hard work for you in this case (plain-RSA key-exchange and server privatekey available). Just go to Preferences / Protocols / SSL and give it the (clear) privatekey file and it will do the key computations and decryption.

If you want to understand a protocol, looking at the specification of that protocol (when available and IETF ones are) is usually a good idea. Note that SSLv3 is showing its age, most recently with POODLE (see several dozen recent questions), and you really should use TLSv1.1 or better (also addressing BEAST). Under ClientKeyExchange we find:

5.6.7.1. RSA Encrypted Premaster Secret Message 


   If RSA is being used for key agreement and authentication, the client
   generates a 48-byte premaster secret, encrypts it under the public
   key from the server's certificate or temporary RSA key from a server
   key exchange message, and sends the result in an encrypted premaster
   secret message.

        struct {
            ProtocolVersion client_version;
            opaque random[46];
        } PreMasterSecret;

   client_version:  The latest (newest) version supported by the client.
      This is used to detect version roll-back attacks.

   random:  46 securely-generated random bytes.

So your RSA-decrypted value is the premaster secret, and is for SSLv3 although you didn't say so. For any key-exchange once the premaster secret is established, you use it along with both Hello.random values to derive (compute) a master secret as described in 6.1. Asymmetric Cryptographic Computations and then use the master secret with the nonces (which will be new nonces in the case of session resumption) to derive the working keys and if applicable IVs, plural, as described in 6.2.2. Converting the Master Secret into Keys and MAC Secrets.

Note in TLSv1(.0) and 1.1 the key derivation functions are changed to better mix MD5 and SHA1, the first more clearly named 8.1. Computing the Master Secret and the second moved forward to 6.3. Key calculation but both share a PRF moved to 5. HMAC and the Pseudorandom Function. Further in TLSv1.2 the KDFs are changed to use SHA-2. Also note the "export" ciphersuites used a modified KDF in SSLv3 and TLSv1(.0); in 1.1 and 1.2 these suites are officially deleted and the modified KDF with them, but some implementations may still allow them (but if they do you shouldn't use them).