Where to store AES keys when encrypting files at rest?

Question

I've gone through the below question:

• How to store an AES Key? an Initialization Vector(IV)?

And read the below articles about how to encrypt files using AES in CBC mode:

• http://www.novixys.com/blog/using-aes-encryption-decryption-python-pycrypto/
• https://eli.thegreenplace.net/2010/06/25/aes-encryption-of-files-in-python-with-pycrypto

But didn't get my answer. Instead of the file assume that I have text for encryption so I do as below:

from Crypto.Cipher import AES
import hashlib

password = 'A_VERY_STRONG_RANDOM_PASSWORD'
key = hashlib.sha256(password).digest()

IV = ''.join(chr(random.randint(0, 0xFF)) for i in range(16))

mode = AES.MODE_CBC
encryptor = AES.new(key, mode, IV=IV)

text = 'j' * 64 + 'i' * 128
ciphertext = encryptor.encrypt(text)

Users of the system has hundreds of files and they all need to be encrypted at rest, so I want to use AES as its encryption is fast and secure.

The first question is that do I need to generate a new password everytime I want to encrypt a file? Or should it be per user? or system wide? Which one is the best practice and more secure?

The second question is that where should I store these/this password(s)?

NOTE: I don't want to use HSM or any external hardware for this.

Answer 1

Let's call the system that performs the encryption E and any system that stores the encrypted files S (for storage).

Files stored on S are secure as long as the attacker doesn't have the corresponding key.

Since you don't want to use an HSM we can assume that any keys stored on E will be compromised if E is compromised. Now you have basically two options for storage: disk or memory. On disk is pretty trivial: you use OS ACL's to restrict access to the encryption key, and hope that it will be fine. For in memory you can build a system that requires you to manually supply a master secret (typing in, read a QR code via webcam, etc.) when the encryption service starts (or on key rotation, see below). This way the key can't be accessed via vulnerabilities that provide FS access only (e.g. path traversal). Note that in Python you will have a hard time erasing secrets from memory, but you can use dirty hacks like this.

(note that Virtualization Based Security can "emulate" extra HW that would have remarkable benefits but I don't think this matches your use-case)

Expecting the compromise of E, it would be a good idea to periodically rotate keys (and move old keys away from E to safe offline storage) so any incident won't affect files encrypted with previous keys. The key rotation period should depend on your needs.

Edit: In case of a decent cipher like AES encryption large number of files with the same key is not a problem, so using a single key doesn't result in weaker encryption than using multiple keys. At the end of the day it's about key management.

The use of per file or per user encrpytion also depends on the key management requirements. Per user keys can make sense if you need to decrypt files on a per-user basis. Per file passwords can limit the impact of a compromise but archiving old keys can be problematic and you should keep in mind that systems usually remain in a compromised state for extended periods of time, so this may not help much more than a sane periodic rekey process.