Should I chunk or not for AES-GCM file encryption

Question

New to file encryption. Need to use AES-GCM for file encryption, can't change the alg. Heard about the max plain text size should be 64GB with same key and IV pair. Environment is python 3 with cryptography module.

Question: As GCM is a stream mode, to encrypt the file should I chunk the file to 128 bits and do encryption on each chunk OR read the whole file at once and pass it to the encryption function (out of memory issue ?) ?

What is the way to read the file and pass it to the encryption function ?

Answer 1

For file encryption it does make sense to stream ciphertext or to use a memory map of a file. Buffering the whole file is definitely not the way to do things.

If you want to use parallelization then you need to split the file into chunks and encrypt / decrypt separately. That will give you authentication tags (and possibly nonces) for each encrypted chunk though. So you would expand your file with e.g. 28 bytes per chunk, rather than 28 bytes per file. Probably you're also going to need a separate file specific key (which can be derived from a master key, if that's required).

---

It is also to actually perform online encryption and decryption for GCM. You can even split encryption and tag creation and verification and decryption of the entire file so that they can be performed in parallel. Of course the latter will mean that you should not use the decrypted file before the verification has actually performed.

Now splitting the ciphertext in chunks is easy once you see that they are just chunks of CTR encrypted plaintext. Putting the separate GMAC of the ciphertext together is a lot harder and requires modular exponentiation.

Probably best start by just encrypting a file with GCM and deliver the ciphertext and plaintext in chunks. Preferably use an API where the authentication tag is considered separate from the plaintext (i.e. provides the online property talked about above).

---

Python is not very fast when it comes to crypto; it mainly requires the use of C-based libraries to perform encryption smoothly. I would suggest that for low level functions such as file encryption, C is actually vastly superior to Python for this kind of functionality (and vastly superior to Java - my fave language - for this kind of usage, in case you think I'm biased).

Answer 2

If you are dealing with text files of size 64 GB or even smaller I believe you should chunk the file and start encrypting portions immediately. This was the former option you mentioned above.

The reason to use GCM vs something like CBC is because GCM can be run in parallel while CBC which is equally secure cannot. If you 'waste' the cpu's ability to start encrypting while it is waiting on the file to load into memory it will be less efficient. GCM is an algorithm meant to be run with " instruction-level parallelism" to refer to the wikipedia page.

You should have the program start loading the file, start encrypting the first 128-bit portion and continue till the whole doc is encrypted.

Python cryptography docs and Pycryptodome docs

Answer 3

AES GCM is a version of AES in counter mode, and a counter mode converts a block cipher into a stream cipher. The main advantage of a stream cipher is that we do not have to worry about padding. We get 16 bytes of pseudo random key stream per invocation of the AES encryption function, which gets XOR ed with the plaintext to get the corresponding ciphertext. When reading a file in programming languages like C or C++ or Python you can specify the number of bytes to read , while looping till you reach the end of file. It is advisable to choose a multiple of 16 in this case like 1024*1024 bytes or more, but keeping hardware limitations and efficiency in mind. Reading 16 bytes for every loop interation is inefficient. You can find an optimal value to read by trial and error, and some benchmarking. The number of bytes successfully read by these functions provided by those programming languages can also be checked (let that be called x). In the case of Python:

# This is only for illustrative purposes!

MY_BLOCK_SIZE = 1024*1024

with open(filename, "rb") as fobj:
     run = True
     while run:
         data = fobj.read(MY_BLOCK_SIZE)
         x = len(data)
         if x == 0:       
               break
         elif (x%MY_BLOCK_SIZE) == 0:
              #invoke AES encryption function MY_BLOCK_SIZE/16 times incrementing the counter each time
             # do all the things for calculating the authentication tag
              # Ciphertext is XOR(data, key_stream, len(data))
         else:
             #invoke AES encryption function MY_BLOCK_SIZE/16 times incrementing the counter each time # do all the things for calculating the authentication  # Ciphertext is XOR(data, key_stream, len(data))

Suppose you read a file of arbitrary length in blocks of bytes (which is a multiple of 16 ) , there are two possibilities, either the last block read might be exactly your block size (x = 16*n) or less than it (x < 16*n). In either case, you have to invoke the AES encryption function n times, incrementing counter for every n, but you use only, x bytes of the key stream, generated for that iteration, to XOR with the plaintext.