1

I have written a piece of software that symmetrically encrypts big files (multi-megabyte). Plaintext is encrypted, after the encryption the HMAC is calculated and written into the header of the file. During encryption, I update HMAC continuously, using as input chunks of encrypted data (encrypt then MAC). There are no checksums on each chunk, so the only possibility to check whether encrypted file was compromised is read whole encrypted file, calculate HMAC of the whole file and compare it with the value stored in the header of the encrypted file.

My question: on decryption, is it better to authenticate encrypted data first and then attempt to decrypt, or proceed with decryption and at the end compare HMAC from the file header with actual HMAC of the encrypted data?

Here are my considerations:

1. Calculating HMAC before decryption attempt seems to me, at first glance, more "secure". On the other hand, I have to read whole file first, calculate and compare HMAC, and then read whole file ONCE AGAIN - this time to decrypt plaintext. This decreases performance - especially if input file is big, but we do not store possibly compromised decrypted plaintext on disk (or anywhere else), so if plaintext is compromised, general user can't access it AT ALL.

2. The other option is to decrypt file straight away, write decrypted output on disk, after whole file is processed calculate HMAC - if it is not equal to the HMAC value from file header, return "file corrupted" error and delete decrypted output from disk (since we consider it invalid). This mechanism increases overall speed of decryption process, since we read input file only once. On the other hand, if input file is compromised, we have written decrypted data on disk unnecessary - because, since HMAC comparison failed, they should be (and will be) deleted after HMAC check.

Does writing decrypted data from potentially tampered encrypted file compromises the security? How would it help to an attacker? As we suppose that the attacker know everything except password, he/she would be able to decrypt compromised file as described in option 2. But I am unable to tell whether this would help him somehow.

Otherwise, we suppose that secure, very long randomly generated passwords are used for encryption, the cipher used is also secure, the password authentication is done via KDF (PBKDF2, scrypt, whatever), so, as decribed above, my only concern is how to perform authentication properly.

EDIT: Please, do NOT answer like "you must first encrypt, then HMAC". The question is NOT about this. I always encrypt, then HMAC. The question is - should I HMAC, then decrypt or vice versa?

Thank you for your attention.

Improve this question
12
  • I think you should HMAC then decrypt if the HMAC is correct. You will be able to prevent some attacks on the implementation. (Am I on the right question this time ? :))
    – r00t
    Jun 24, 2015 at 13:43
  • "vice versa" would be silly. If you're going to write "decrypted data on disk unnecessary", then you might as well be computing the MAC along the way.
    – user49075
    Jun 24, 2015 at 13:44
  • Too bad I have deleted my answer to you comment before. OK, once again - you can't "compute MAC along the way". You can update MAC along the way - and then, when all data to be authenticated, are passed to the MAC - and only then, you can compute MAC. You can't authenticate part of the message. You can authenticate only whole message. And I don't see why "vice versa" would be silly. If provided password is correct, I can decrypt file. But I don't know whether encrypted file is compromised or not. For that purpose, I need to read the whole file, and authenticate it.
    – Acetylator
    Jun 24, 2015 at 13:54
  • The only question is - since I am reading encrypted file anyway, should I simultaneously decrypt it and store (possibly compromised) decrypted data on disk? Or should I first check whether encrypted data are not compromised and then read whole encrypted file again - this time to decrypt it? Reading whole file TWICE decreases performance, this is clear, but does it really increase security? This is my question.
    – Acetylator
    Jun 24, 2015 at 13:57
  • 1
    Yes, for me it increase security. If you have an implementation error in the decryption and the encrypted file is modified to trigger it, you will prevent this by doing the two steps solution.
    – r00t
    Jun 24, 2015 at 14:07

1 Answer 1

Reset to default
2

Obligatory disclaimer: If at all possible, don't implement your own crypto.

Verifying the validity of the decrypted file first and then decrypting it regardless of whether the MAC matches is the most secure way of doing this. After this, make sure that if any error occurs the same kind of error is always returned.

Disclosing any information on the reason of a failure is a serious security risk. For example if a bad MAC and bad padding in the decrypted file return different error messages a padding oracle attack can be mounted. Some padding oracle attacks, like lucky thirteen, only need to notice a difference in time in order to mount an attack on your crypto. This is why it's safest to always also decrypt the file regardless of MAC mismatch.

Depending on the crypto primitive and the mode of operation you are using different attacks should be considered.

Improve this answer
5
  • I understand that MAC-then-decrypt should be more secure than decrypt-then-MAC. However, I would like to know whether decrypt-then-MAC brings any real security issues, since I try to improve performance - but, obviously, not for the price of security. In my software, there are only two types of error messages - "invalid password" and "file is corrupted". Checking password is computationally expensive (done with KDF), and you can't skip it since it provides entropy for generating cipher key.
    – Acetylator
    Jun 24, 2015 at 14:51
  • I fail to see how decrypt-then-MAC performs worse? What modes and ciphers are you using?
    – Juha Kivekäs
    Jun 24, 2015 at 15:49
  • I use AES-256 in CBC mode. MAC-then-decrypt performs worse, because I have to read whole file (without decryption),calculate MAC of the encrypted text and compare with it with the MAC saved in the file header. If both MACs are the same, then I have to read whole file once again - this time to decrypt it. So the whole file will be read from disk 2 times.If I do decrypt-then-MAC, I read file from disk only once - performing decryption simultaneously with updating MAC.After decryption is finished and decrypted data are written on disk,I check whether MACs match. If not,decrypted file is deleted
    – Acetylator
    Jun 24, 2015 at 18:09
  • Lucky13 depends on MAC-then-encrypt; the tampered padding alters the length of the cleartext to be verified across an input-block boundary, because decrypt is (must be) done before verify. One obvious way to avoid any padding oracle is to use a stream mode (like CTR) or cipher (but not RC4 please). Or an authenticated mode like GCM or CCM eliminates the which-first issue. But even for CBC + HMAC, I think if you do only the blockwise decryption (constant-time in ciphertext length) concurrent with HMAC and if good then the unpadding you're okay.
    – dave_thompson_085
    Jun 25, 2015 at 23:40
  • As an offside note: while lucky13 does depend on MAC-then-encrypt, generally padding oracles don't depend on it.
    – Juha Kivekäs
    Jun 26, 2015 at 3:31

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .