Encrypting/wrapping a private RSA key in PKCS8 using AES-GCM and openSSL 3.20 (library not command line tool)

Question

I need to protect a private RSA key using a passphrase but using AES-GCM for the actual encryption. This is normally done using various a combination of openSSL library calls. However, I now need to support using AES-GCM instead of AES-CBC that is usually used. But it fails with following error:

error:0300007A:digital envelope routines::cipher parameter error

The relevant code

const EVP_CIPHER*  cipher = EVP_get_cipherbyname(SN_aes_256_gcm);
X509_ALGOR*        pbe = PKCS5_pbe2_set_iv(cipher, 64000, nullptr, 0, nullptr, NID_hmacWithSHA256);

The error occurs at the PKCS5_pbe2_set_iv, pbe gets set with a nullptr.

Is it remotely possible to ever think that openSSL 3.2 can do AES-GCM for this operation? GCM is said to be better than CBC....

Answer 1

GCM allows authentication of data, but it depends on how it is used if it is more secure for key wrapping operations. If no salt or IV is given then using the same password will result in a very insecure system as:

1. the GMAC-generated authentication tag is calculated using a derived key that is insecure in the given context;
2. the CTR mode of encryption that is used within GCM can lead to loss of information if the ciphertext are XOR'ed against each other.

You're better off using either:

1. the original CBC, preferably with a random IV if you don't want to leak any data about the key;
2. a specific key wrapping algorithm, usually called AES-wrap;
3. AES in (GCM-)SIV mode, because this includes all the information of the wrapped key in the calculation of the IV and therefore authentication tag.

The disadvantage of CBC is of course that it doesn't add an authentication tag. Usually the idea is that confidentiality is enough, so that's often OK. An authentication tag can be useful if you're not sure that the wrapped key cannot be altered. However, please remind yourself that this doesn't protect against deletion or e.g. supplanting the key with another key wrapped with the same password.

So uh, I guess my answer is: don't.

Answer 2

I decided to do it myself, using the low-level functions available in OpenSSL.

The longest part to develop was the asn1 string generation.