New to file encryption. Need to use AES-GCM for file encryption, can't change the alg. Heard about the max plain text size should be 64GB with same key and IV pair. Environment is python 3 with cryptography module.

Question: As GCM is a stream mode, to encrypt the file should I chunk the file to 128 bits and do encryption on each chunk OR read the whole file at once and pass it to the encryption function (out of memory issue ?) ?

What is the way to read the file and pass it to the encryption function ?

  • 1
    python cryptography's API, like most, uses an init, update..., finalize structure that allows you to process data in chunks of any size you like and get the correct result. Using very small chunks is likely to be much less efficient though; I'd start with a megabyte at a time, and measure. Yes, you should not exceed 64G total (with the default tag size of 128 bits). – dave_thompson_085 Nov 17 '19 at 2:39

If you are dealing with text files of size 64 GB or even smaller I believe you should chunk the file and start encrypting portions immediately. This was the former option you mentioned above.

The reason to use GCM vs something like CBC is because GCM can be run in parallel while CBC which is equally secure cannot. If you 'waste' the cpu's ability to start encrypting while it is waiting on the file to load into memory it will be less efficient. GCM is an algorithm meant to be run with " instruction-level parallelism" to refer to the wikipedia page.

You should have the program start loading the file, start encrypting the first 128-bit portion and continue till the whole doc is encrypted.

Python cryptography docs and Pycryptodome docs

|improve this answer|||||
  • Thanks for the answer. My doubt in that case is If the operation need to be done block wise then why is it called the stream cipher ? – 3lokh Nov 17 '19 at 1:57
  • 1
    The encryption and decryption parts of GCM can be parallelized, but both generate and verify GMAC can't, and the security rules say you must verify before decrypting (although people don't always obey the rules, cf efail). CBC doesn't provide integrity like GCM, or AAD either, although you can add both. Any crypto can overlap I/O in principle, although python's global lock greatly limits parallelism. – dave_thompson_085 Nov 17 '19 at 2:36

For file encryption it does make sense to stream ciphertext or to use a memory map of a file. Buffering the whole file is definitely not the way to do things.

If you want to use parallelization then you need to split the file into chunks and encrypt / decrypt separately. That will give you authentication tags (and possibly nonces) for each encrypted chunk though. So you would expand your file with e.g. 28 bytes per chunk, rather than 28 bytes per file. Probably you're also going to need a separate file specific key (which can be derived from a master key, if that's required).

It is also to actually perform online encryption and decryption for GCM. You can even split encryption and tag creation and verification and decryption of the entire file so that they can be performed in parallel. Of course the latter will mean that you should not use the decrypted file before the verification has actually performed.

Now splitting the ciphertext in chunks is easy once you see that they are just chunks of CTR encrypted plaintext. Putting the separate GMAC of the ciphertext together is a lot harder and requires modular exponentiation.

Probably best start by just encrypting a file with GCM and deliver the ciphertext and plaintext in chunks. Preferably use an API where the authentication tag is considered separate from the plaintext (i.e. provides the online property talked about above).

Python is not very fast when it comes to crypto; it mainly requires the use of C-based libraries to perform encryption smoothly. I would suggest that for low level functions such as file encryption, C is actually vastly superior to Python for this kind of functionality (and vastly superior to Java - my fave language - for this kind of usage, in case you think I'm biased).

|improve this answer|||||

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.