How do I create a valid email certificate for Outlook S/MIME with openssl?

Question

I need to create a certificate for email encryption and signing that has to be used by Outlook 2003+. I'm using OpenSSL, my self-signed root-CA is already imported into the trusted root-CA store. These are my steps to create a p12 Identity file importable by Outlook:

openssl req -batch -newkey rsa:1024 -keyout KEY.key -out KEY.csr \
  -nodes -config openssl.cnf &&\
openssl x509 -req -sha1 -days 1000 -in KEY.csr -CA ca.crt -CAkey ca.key \
  -set_serial 1 -out KEY.crt -setalias "FRIENDLY_NAME" \
  -clrtrust -addtrust emailProtection \
  -addreject clientAuth -addreject serverAuth -trustout &&\
openssl pkcs12 -export -in KEY.crt -inkey KEY.key" -out KEY.p12 \
  -name "FRIENDLY_NAME" -passout pass:PASSWD &&\
chmod 0600 KEY_CN.{key,p12} &>/dev/null

Here is the relevant segment of my openssl.cnf:

[ usr_cert ]
basicConstraints = CA:FALSE
authorityKeyIdentifier = keyid
subjectKeyIdentifier = hash
keyUsage = critical, digitalSignature, keyEncipherment
extendedKeyUsage = critical, emailProtection
subjectAltName = email:copy
authorityKeyIdentifier = keyid
subjectKeyIdentifier = hash

I inspected another valid p12 file to get these settings - and the -name cli option in the 3rd openssl-statement above.

My problem: Outlook still chokes on this (it's valid in Thunderbird, though). I don't have the English error message (could provide the German one, though), these are the rough translations of the dialogs I see:

1. "Grant usage of a key to the application: Grant / do not grant" (you may show key details in the lower left - it says there is no description and no context information)
2. "Repeat the procedure. The protected key cannot be accessed, make sure the specified password is valid." (I did - I even tried every possible variation of password/no password in openssl and Outlook! For the next step, I'm led back to the first dialog and click "Cancel")
3. "Error in the underlying security system. Access denied!"

My current setup in Outlook: Sign emails, transfer text and signature in plaintext and attach the certificate. I also restricted the certificates usage to secureEmail and disabled OCSP. When importing the certificate (and on generation) I used the exact same password for every password request and the same description as friendlyName, CN and alias (maybe these can be different, I just didn't want to risk anything while trying).

I really don't get what Outlook doesn't like / wants to tell me. If needed, I will gladly provide additional details.

Thanks!

Answer 1

The dialog box in #1 is expected when operating under Medium security. The OS is supposed to verify the PIN before using the key for the first time.

The dialog boxes in #2 and #3 indicate more of a file-based permissions issue. Are you doing this in Cygwin under Windows or a Unix host and transferring the file over? There might be a problem in the chmod at the end under Cygwin.

Consider removing the "-clrtrust -addtrust emailProtection -addreject clientAuth -addreject serverAuth -trustout" options and seeing if it works when all intended uses are allowed. Then start scaling back the intended uses.

For example, the following works for me, but obviously has "all" intended uses which may not be desirable:

$ openssl genrsa -aes128 -out email.key 2048
$ openssl req -new -key email.key -out email.csr -config email.cnf
$ openssl x509 -req -days 365 -in email.csr -CA ca.crt -CAkey ca.key -set_serial 10 -out email.crt
$ openssl pkcs12 -export -in email.crt -inkey email.key -out email.pfx

I use a very basic cnf file:

[ req ]
default_bits           = 2048
distinguished_name     = req_distinguished_name
prompt                 = no


[ req_distinguished_name ]
C                      = {Country}
ST                     = {Provice/State}
L                      = {City}
O                      = {Org}
OU                     = {Org Unit}
CN                     = [email protected]
emailAddress           = [email protected]

Load the email.pfx into the Windows Certificate Manager into your Personal certificate store. Now, I have Outlook 2010 running under Windows 7, so I have to also create the entry I will use for default security settings. This involves going to the Outlook Trust Manager (File | Options | Trust Center | Trust Center Settings | Email Security). The "Encrypted Email" grouping is where you can create a new "Default Setting" which includes the newly imported certificate. (The "Import/Export" button in Outlook 2010 can be used as a way to import the .pfx into the cert mgr.)