Compatibility between OpenSSL 0.9.8r and OpenSSL FIPS Object Module v1.2?

Question

I have a private key in PEM format that was generated by OpenSSL FIPS Object Module v1.2 using a proprietary third party tool (that I cannot avoid using). This third party tool encrypted the private key with a passphrase that it provides to me in base64 format. For example, I generated a dummy certificate, and it tells me that the passphrase is CPV6TJwm5cMd1WeVupo2Lg==.

My goal is to use this private key in a script that connects to an HTTPS server and presents a client certificate for authentication. Therefore, I need to remove the passphrase from the key. I have tried a variety of things, but I cannot seem to decrypt this private key. Here's are two attempts that do not work:

/Users/mhaase/Downloads $ openssl rsa -in FoundstoneClientCertificate.pem -out FoundstoneClientCertificate2.pem -passin pass:CPV6TJwm5cMd1WeVupo2Lg==
unable to load Private Key
59915:error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt:/SourceCache/OpenSSL098/OpenSSL098-47/src/crypto/evp/evp_enc.c:330:
59915:error:0906A065:PEM routines:PEM_do_header:bad decrypt:/SourceCache/OpenSSL098/OpenSSL098-47/src/crypto/pem/pem_lib.c:428:

/Users/mhaase/Downloads $ echo "CPV6TJwm5cMd1WeVupo2Lg==" | openssl base64 -d | openssl rsa -in FoundstoneClientCertificate.pem -out FoundstoneClientCertificate2.pem -passin stdin
unable to load Private Key
59909:error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt:/SourceCache/OpenSSL098/OpenSSL098-47/src/crypto/evp/evp_enc.c:330:
59909:error:0906A065:PEM routines:PEM_do_header:bad decrypt:/SourceCache/OpenSSL098/OpenSSL098-47/src/crypto/pem/pem_lib.c:428:

I found on a message board that this third party program doesn't use normal OpenSSL; it uses OpenSSL FIPS Object Module v1.2, which is a separate-but-related piece of software that has undergone U.S. federal testing and certification. So I went back to the server that generated this key and I ran openssl rsa -in foo.pem -out bar.pem and it asked for the passphrase, I pasted the base64 passphrase, and it worked!

According to the OpenSSL FIPS Object Module v1.2 User Guide:

The FIPS Object Module provides an API for invocation of FIPS approved cryptographic functions from calling applications, and is designed for use in conjunction with standard OpenSSL 0.9.8 distributions beginning with 0.9.8j. Note: OpenSSL 1.0.0 is not supported for use with the OpenSSL FIPS Object Module. These standard OpenSSL 0.9.8 source distributions support the original non-FIPS API as well as a FIPS mode in which the FIPS approved algorithms are implemented by the FIPS Object Module and non-FIPS approved algorithms other than DH are disabled by default. These non-validated algorithms include, but are not limited to, Blowfish, CAST, IDEA, RC-family, and non-SHA message digest and other algorithms.

This indicates to me that it should be no problem for one version to decrypt keys that were encrypted by the other version. But my experience indicates otherwise. Am I missing something?

Answer 1

Firstly can you please check that in the initial PEM file it contains actual RSA private key. If you cat the contents of the file it should say something like this:

-----BEGIN RSA PRIVATE KEY-----

I highly suspect that the file you think is the private key does not actually contain a private key.

Depending on how the original system was using the certificate they may be a p12 file (certificate + chain and private key) somewhere which you could extract the key from. But if this was taken from a system running Apache then Admins and Devs like to give the key a .key extension and this would just be the actual RSA key not a p12.

As a side note I also suspect that the passphrase to the key is not the Base64 encoded version but a the actual decoded string which in this case is

zL&g6.

If you do find a .key file then you don't need to do anything. You should be anble to issue the openssl s_client command and do a test connection to the server using client auth e.g.

openssl s_client -connect host:443 -cert cert_and_key.pem -key cert_and_key.pem 

If you do need to extract the private key from a p12 please refer to this for further help: https://stackoverflow.com/questions/9497719/how-to-extract-a-public-private-key-from-a-pkcs12-file-with-openssl-for-later-us