SMIME email decryption key with OpenSSL

Question

I'd like to know if it's possible to use openssl command to retrieve and decrypt the key for encrypting/decryptig email content. I know I can decrypt the whole encrypted mail by something like this

openssl smime -decrypt -in enc_mail.eml -inkey recip_priv.pem > dec_mail.eml

However, I'd like to see the steps in between. If I understand the procedure right, the actual content of the email is being encrypted not by the recipient's pubkey, but with randomly generated key on sender's side. This key is then encrypted with recipient's pubkey and attached to the encrypted message. Am I right? Is it possible to use openssl to show me the attached encrypted key and decrypt it separately?

Thanks.

Answer 1

Yes you can. This example uses openssl smime with the default RC2 CBC with a 40-bit key. The newer cms sub-command behaves slightly differently, and uses 3-DES by default. You probably shouldn't be using either of those algorithms to encrypt important data ;-)

There are two minor caveats: firstly, I'm also going to use a couple of other tools (though OpenSSL is used for all the heavy lifting), and secondly I'm going to make some assumptions about how the email was encrypted.

The en/decryption is along the lines of most RSA-using methods: use (slow, expensive) RSA to en/decrypt a symmetric key, and use the fast symmetric key to en/decrypt the real data. (See this question or this for more background).

Take your email, extract the P7M part, and decode it. If you have a single .p7m part that is base64 encoded you can do this easily with metamail:

$ metamail -wy enc_mail.eml

Save the P7M file. This is an ASN.1 DER encoded CMS (PKCS#7) file, so we can peek inside:

  $ dumpasn1 -tilda  smime.p7m
   0 1946: SEQUENCE {
   4    9: . OBJECT IDENTIFIER envelopedData (1 2 840 113549 1 7 3)
         : . . (PKCS #7)
  15 1931: . [0] {
                  [ .. certificate details and whatnot omitted ...]
 188   13: . . . . . SEQUENCE {
 190    9: . . . . . . OBJECT IDENTIFIER rsaEncryption (1 2 840 113549 1 1 1)
         : . . . . . . . (PKCS #1)
 201    0: . . . . . . NULL
         : . . . . . . }
 203  256: . . . . . OCTET STRING    
         : . . . . . . A0 DA EA FB EA 1A 0F 81    ........
         : . . . . . . F4 30 9F 78 5C 9B A7 27    .0.x\..'
                       [ ... blob snipped ...]
 463 1483: . . . SEQUENCE {
 467    9: . . . . OBJECT IDENTIFIER data (1 2 840 113549 1 7 1)
         : . . . . . (PKCS #7)
 478   26: . . . . SEQUENCE {
 480    8: . . . . . OBJECT IDENTIFIER rc2CBC (1 2 840 113549 3 2)
         : . . . . . . (RSADSI encryptionAlgorithm)
 490   14: . . . . . SEQUENCE {
 492    2: . . . . . . INTEGER 160
 496    8: . . . . . . OCTET STRING 3E EA 0E 12 37 A8 56 70                 
         : . . . . . . }
         : . . . . . }
 506 1440: . . . . [0]    
         : . . . . . A5 FF A1 70 2C AD 82 6A    ...p,..j
         : . . . . . C7 F0 84 E8 9E 93 8F 53    .......S
                     [... blob snipped ...]

I'm using dumpasn1 because openssl asn1parse is disinclined to display or dump the various blobs we're interested in. The structure of your email will vary from the above of course. Columns 1 and 2 are the offset and size of each (possibly nested) sub-structure.

The interesting parts are at offsets:

• 188 the RSA encryption details, followed at offset 203 by encrypted data
• 463 the S/MIME encryption details and parameters (RC2 CBC)
• 506 the encrypted blob

At offset 188 we can see RSA is used, followed by 256 bytes of data, so extract that data blob (offset 203) and convert it to binary:

$ dumpasn1 -a -203 smime.p7m | tail -qn +2 | xxd -r -p > rsa.bin

(Note starting offset -203 and the use of tail to skip over the first line of output. This is a little convoluted, but unfortunately both dumpasn1 and openssl asn1parse leave the type-length prefix intact when you try to slice and dice objects.)

Decrypt this data using RSA:

$ openssl rsautl -inkey recip_priv.pem -in rsa.bin -decrypt -out rc2key.bin
$ xxd -u -p  rc2key.bin
92F6EB53B1

In this case we get 5 bytes (40-bit) output, the symmetric key we need. The input was PKCS#1 v1.5 padded (see §8.1) for various reasons, hence the size disparity.

The main payload (email) is in the blob at offset 506, extract that to a file:

$ dumpasn1 -a -506 smime.p7m | tail -qn +2 | xxd -r -p > email.bin

Now here's the slightly tricky bit, for RC2 refer to section 6 of RFC 2268:

rc2CBC OBJECT IDENTIFIER
 ::= {iso(1) member-body(2) US(840) rsadsi(113549)
      encryptionAlgorithm(3) 2}

RC2-CBCParameter ::= CHOICE {
  iv IV,
  params SEQUENCE {
    version RC2Version,
    iv IV
  }
}

RC2Version ::= INTEGER -- 1-1024
IV ::= OCTET STRING -- 8 octets

This explains the data structure at offset 490:

 480    8: . . . . . OBJECT IDENTIFIER rc2CBC (1 2 840 113549 3 2)
         : . . . . . . (RSADSI encryptionAlgorithm)
 490   14: . . . . . SEQUENCE {
 492    2: . . . . . . INTEGER 160
 496    8: . . . . . . OCTET STRING 3E EA 0E 12 37 A8 56 70                 
         : . . . . . . }

(You can confirm that RC2Version 160 (0xa0) matches key size 40 bit (0x28) in the EKB table.)

So, putting it all together: the S/MIME encryption algorithm (RC2 40 bit CBC, offset 480), the RC2 key (decrypted from blob at offset 203), the RC2 IV (not encrypted, offset 496) and the encrypted payload (offset 506):

$ openssl enc -d -rc2-40-cbc -in email.bin -out email.txt -K 92F6EB53B1 -iv 3EEA0E1237A85670

and email.txt should be what you're looking for.

Tips:

• make sure not to use an ancient version of xxd, it may mangle input hex data
• the structure, sizes and offsets will of course vary by message, keys and algorithm
• I find openssl enc -kfile ... does not work, stick to -K
• openssl smime may have modified your message before encryption (CRLF requirements)
• berdump is a handy tool as its output is more amenable to further processing. Since DER is a subset of BER, you can point it directly at a PKCS#7 DER file

Answer 2

S/MIME uses PKCS7. You probably have to write the code yourself.

Here is an example how to do that in C++: https://stackoverflow.com/questions/6369096/how-to-read-the-certificates-file-from-the-pkcs7-p7b-certificate-file-usind-open

And this is an example for Java: https://stackoverflow.com/questions/11026588/pkcs7-encryption