4

I am storing files on S3, some of which are very large, and need to be stored encrypted. I had planned on using Galois Counter Mode since it includes the MAC, however this won't work with multipart uploads to S3. I need to encrypt each chunk and store it on S3 along with a proper MAC, then calculate the MAC for the entire file once all parts are in place. Once a multipart upload to S3 is "completed" the parts are no longer accessible and there is just the single file. I think I need some tree hash mechanism for supporting this (note that the Amazon Glacier storage system is using this method), but I would rather avoid rolling my own MAC for all the usual reasons.

Here is a page discussing how Glacier uses tree hashing: http://docs.amazonwebservices.com/amazonglacier/latest/dev/checksum-calculations.html

Any suggestions? Many thanks in advance.

Update

I finally got this working. I hacked up the GCMBlockCipher class in BouncyCastle to support this. Here are the changes that need to be made:

  1. The encryption key and IV will need to be safely stored somewhere (perhaps encrypted in a database) so that all machines processing the individual parts will be able to calculate the same initial values like hash subkey.
  2. While the total length does not need to be known in advance, there will need to be a starting block index sent with each chunk, and each chunk sent (except final) must be a multiple of 16 bytes.
  3. The counter value needs to be incremented based on the starting block index for the chunk.
  4. An implementation that handles encryption as one long stream performs a single GHASH multiplication per processed block. By the time the last block is processed the hash value will be correct. In a parallel implementation you may be processing a chunk somewhere in the middle, so you have to make up for the "missing" multiplications that would have been performed when processing subsequent blocks. If the total length is known in advance this correction can be applied right after each chunk is processed, or at the end once all parts are in place. The hash values for each chunk will need to be stored.
  5. At the end all of the hash values for the chunks (with multiplication corrections applied) need to be xor'd together then fed into a modified form of doFinal() which performs the rest of the GMAC calculations. Note that the final byte count needs to be fed in as well since that is part of the MAC.

The more mathematically inclined can probably come up with a better way of handling this. :-)

Improve this question
5
  • This seems to be asking about "data integrity over transport" which doesn't have any security implications per-se and so it may be considered off topic. A better spot for this may be programmers.SE, or serverfault, or StackOverflow.
    – makerofthings7
    Dec 19, 2012 at 21:48
  • 3
    I'm still not sure how S3's transport mechanism prevents you from encrypting the file with AES-GCM beforehand. Why would you need to verify the data while before it's been reassembled? Just encrypt, split into appropriately-sized chunks, upload individually, and the file should match the original once the parts have been recombined.
    – Stephen Touset
    Dec 19, 2012 at 22:36
  • This is about the data being secure at rest and not readable by Amazon staff, or readable by others if an incorrect ACL allows the file to be downloaded. The encryption keys are stored on our systems and not accessible by clients, so encrypting the file in its entirety before client sends to us is not an option. The parts that the client sends can arrive at any of a number of production servers, so encrypting the entire file server side is not an option either. So as I receive each file chunk I need to encrypt and store it, and I got stuck on how to handle the MAC.
    – Peter Friend
    Dec 19, 2012 at 22:50
  • When verifying, do you want to be able to work on pieces, or is working on the whole file enough?
    – CodesInChaos
    Dec 20, 2012 at 10:45
  • @CodesInChaos, I want to be able to validate each part as it comes in, and reject the part if found to be corrupt. Once all parts are in place I want to combine the individual MACs to produce a MAC for the entire file. When the uploads are complete only the whole file MAC will be needed as we don't currently have a need for verifying chunks of a file as it is downloaded.
    – Peter Friend
    Dec 20, 2012 at 19:44

1 Answer 1

Reset to default
1

The GHASH component of AES GCM is linear. If you know the lengths of all file parts and the order in which they will be assembled, you can use this linearity to compute the polynomial value you need, and then add in the random part that makes GHASH into GCM. The spec for GCM should provide enough guidance on how to do it.

Improve this answer
2
  • Thanks, this also seems to be what this paper is saying: mindspring.com/~dmcgrew/gmac-incr-c-00.pdf. I'll read the spec again and try to figure it out.
    – Peter Friend
    Dec 20, 2012 at 2:30
  • Looks like it will work. Each part uploaded (except last) will need to be a multiple of 128 bits, and I will need the part number. This should allow me to compute the CTR block (assuming that the IV and some other details are stored somewhere). Still figuring out the GHASH details. Will update this post once I have a working prototype.
    – Peter Friend
    Dec 21, 2012 at 2:20

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .