1

Relevant question for Python:

  1. Stack Exchange: https://stackoverflow.com/questions/75739308/aead-authentication-with-huge-input-that-doesnt-fit-into-ram
  2. Top Answers: https://topanswers.xyz/python?q=3867

Basically: data that doesn't fit into main memory needs to be authenticated with AEAD (associated with a tiny bit of data that also needs to be encrypted), so I'm wondering whether it would be OK to instead hash the data and then just authenticate the digest. Are these two options equivalent? Are there any weird implications?

Improve this question
4
  • 1
    GCM has the advantage that GMAC is much faster than a generic hash function. If you use your trick you might as well use CTR + HMAC over a canonical representation of the IV, AAD and data.
    – Maarten Bodewes
    Mar 15 at 12:54
  • 1
    Another method is to use multiple sections and use AES-GCM on those, then calculate the HMAC over the result. You could even use AES-GCM with the tags as AAD. With these schemes you have to be careful not to overstretch the nonce though; calculating a derived key for each file would be a good idea. This kind of scheme allows for parallel decryption & verification.
    – Maarten Bodewes
    Mar 15 at 13:17
  • @MaartenBodewes In "use multiple sections and use AES-GCM on those, then calculate the HMAC over the result", what is the "result"? In "With these schemes you have to be careful not to overstretch the nonce though; calculating a derived key for each file would be a good idea.", what do you mean "for each file", I only have a single file? Did you mean "for each section"? When you say section, do you mean to split my data into sections logically and then encrypt them separately?
    – user2373145
    Mar 15 at 18:05
  • 1
    Sorry, I meant "over the tags". As for the key derivation: you can e.g. use an increasing nonce to encrypt each part. However, if you encrypt many keys with the same key then the 12 bit nonce may become a problem, as the chance of collisions is quickly increasing. That can be avoided by deriving a different key.
    – Maarten Bodewes
    Mar 15 at 18:11

1 Answer 1

Reset to default
1

Provided you use a cryptographically secure hash function, then yes, this is secure. This is because any attempt to modify the additional data would modify the hash with high probability, and if the hash were modified, the tag would fail to verify with high probability. It can be thought of somewhat related to a Merkle tree. SHA-2, SHA-3, and BLAKE2 are all fine options here; MD5 and SHA-1 are not.

In many AEADs, the associated data portion is online, just like the encrypted portion, so assuming the library interface is suitable, you shouldn't need to do this. However, not all library interfaces allow streaming additional data, and some AEADs (e.g., CCM) are not online at all, and so this can be useful if you need a very large or unknown amount of additional data.

Note that if you have a small amount of fixed additional data, such as a packet header, you can also pass that data, followed by a hash of the large quantity of data, as the AEAD's additional data, if that's more convenient. However, to avoid attacks where the attacker moves data between the hash and the packet header, you must always use the hash, even if the "large" quantity of data is empty, and use the same hash algorithm with the same size output.

In general, it's important to note whenever your additional data contains multiple segments that you must avoid cases where the attacker can confuse which part is which or move data between the parts, such as by prefixing each segment with a 4- or 8-byte length of its segment, or always using fixed-length data segments. This is true whether you pass the AD to a hash algorithm or use it directly in the AEAD.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.