I am storing files on S3, some of which are very large, and need to be stored encrypted. I had planned on using Galois Counter Mode since it includes the MAC, however this won't work with multipart uploads to S3. I need to encrypt each chunk and store it on S3 along with a proper MAC, then calculate the MAC for the entire file once all parts are in place. Once a multipart upload to S3 is "completed" the parts are no longer accessible and there is just the single file. I think I need some tree hash mechanism for supporting this (note that the Amazon Glacier storage system is using this method), but I would rather avoid rolling my own MAC for all the usual reasons.
Here is a page discussing how Glacier uses tree hashing: http://docs.amazonwebservices.com/amazonglacier/latest/dev/checksum-calculations.html
Any suggestions? Many thanks in advance.
I finally got this working. I hacked up the GCMBlockCipher class in BouncyCastle to support this. Here are the changes that need to be made:
- The encryption key and IV will need to be safely stored somewhere (perhaps encrypted in a database) so that all machines processing the individual parts will be able to calculate the same initial values like hash subkey.
- While the total length does not need to be known in advance, there will need to be a starting block index sent with each chunk, and each chunk sent (except final) must be a multiple of 16 bytes.
- The counter value needs to be incremented based on the starting block index for the chunk.
- An implementation that handles encryption as one long stream performs a single GHASH multiplication per processed block. By the time the last block is processed the hash value will be correct. In a parallel implementation you may be processing a chunk somewhere in the middle, so you have to make up for the "missing" multiplications that would have been performed when processing subsequent blocks. If the total length is known in advance this correction can be applied right after each chunk is processed, or at the end once all parts are in place. The hash values for each chunk will need to be stored.
- At the end all of the hash values for the chunks (with multiplication corrections applied) need to be xor'd together then fed into a modified form of doFinal() which performs the rest of the GMAC calculations. Note that the final byte count needs to be fed in as well since that is part of the MAC.
The more mathematically inclined can probably come up with a better way of handling this. :-)