I have a backup of my important data in the cloud with a given size of several gigs. To prevent snooping the data is inside a TrueCrypt container and everything which will be transmitted is the TrueCrypt container (no hidden volume).

Because uploading several gigs with even a fast Internet connection is slow, I have split the container file into pieces of several MByte size. I wrote a script which compares the current container with an instance of the past container (The past container is stored on the local disk to speed up the process) and only the changed pieces are written out and finally transmitted to the cloud.

Works great so far. Changes occuring from backup to backup weekly are in the 5% range of the original file, so updating is no problem.

Now a short time ago I prepared the backup and was wondering why it takes so long. I was puzzled to see that the changes have increased from the normal 5% to 50% ! Alarmed, I mounted both volumes (old one and the current one) and made a file compare and could not see any reason for the increase.

I have no knowledge how exactly TrueCrypt lays out the data internally, but as a file system the simple explanation could be that some data is heavily fragmented and deletion/appending such data could cause such an effect.

But I am still uneasy, so

a) Is the simple explanation correct and it is normal that changing only small portions of data could cause changes affecting a much bigger portion of the data in the TrueCrypt container ?

b) If you think a) is not a sufficient reason and you find that it is strange behavior, how to continue ?

ADDITION: There are several misconceptions here, so I will now describe exactly what I am doing because it does not seem "obvious" enough.

  1. I have several gigs of data. And yes, if you have pictures, documents, your source code repository and other important stuff you really want to have again if e.g. your computer is destroyed, then you easily accumulate several gigs.
  2. I want to store the data in a cloud server over webdav. Because I don't want anyone to snoop in my data, I am using encryption and because I do not distinguish between critical confidentiality and...what exactly is the other option ??... so I use TrueCrypt 7.1a.
  3. TrueCrypt offers the option to mount a single big file as virtual drive. The encryption is the Serpent/Twofish chain (no AES). In the meantime I calculated the hash values of the binaries and TrueCrypt seems to be ok, so one possible problem is eliminated (not strictly, but there is hope)
  4. The TrueCrypt volume is mounted, let's say it has the name TC. The first step of the script is started; it compares the content of specific directories (e.g. /foo) to the equivalent TrueCrypt (TC/foo) and updates the TrueCrypt content so that they are equivalent to the original directory. The update uses simply delete/copy and this is the only step where the data is changed. There is no data corruption !
  5. TrueCrypt volume is unmounted. We are looking now at the file used for the virtual drive. It is large and looks random. But because we have changed the content, its binary content has also changed. It is a fact that can be reproduced: If you have committed only small changes in the virtual TrueCrypt drive (at least under Serpent/Twofish) the binary blob normally has only small changes, too.
  6. Now I want to transmit the data as fast as possible. Because the internet connection is a bottleneck, I always have a copy of the old TrueCrypt file. Because I know now that normally only small parts are changed, the big binary Truecrypt file blob is split into smaller segments in the cloud. When changes occur, only the changed segments are then updated. Naturally because the segments have a constant size, I transmit more data than necessary, but it is ok. The cloud is also handled as virtual drive.
  7. So what did I use as file operations ? Essentially delete/copy/remove/binary compare/split. It's not a programming question. It's in my humble opinion neither a "home-growing solution to hack together things that weren't designed to integrate with each other in the first place" and it definitely is not a "business class" or "enterprise" problem. If anyone is not able to write such a script in less an hour him-/herself...
  8. Gilles has asked a very good question. The percentage of changed segments is 50%. I run a counter and found that 20,25% of the binary content has changed with (approximate) sizes
    less than 10 bytes: 590 000 times
    less than 100 bytes: 5 000 0000 times
    less than 1000 bytes: 11 000 000 times
    less than 10000 bytes: 350 000 times.
    No bigger changes occur.

    The number of segments is 320, by the way. The next step for me will be to find long byte chains and see if I can locate them on another place (so that my suspicion that it isn't change, only relocation, is confirmed).

I do not know the exact security implications, but if some operations on TrueCrypt files can trigger a large change in the data contained when normally only small changes are observed, it could be a possible weakness.

SOLUTION: I looked up when I started the script and in fact I mounted the TrueCrypt volume during a weekly defragmentation session. I tried now to manually defragment the volume and it indicated 0% defragmentation, so I think this is the solution. At least I was right that something had changed my data without my knowledge. Thanks to Michael Karcher.

closed as off-topic by schroeder, Iszi, Xander, TildalWave, gowenfawr Jan 28 '15 at 20:56

  • This question does not appear to be about Information security within the scope defined in the help center.
If this question can be reworded to fit the rules in the help center, please edit the question.

  • This is a program-specific question about its operational behaviour. This isn't an InfoSec question. – schroeder Jan 28 '15 at 18:02
  • @schroeder Interesting. How do you know that the problem is a program-specific question and definitely not an InfoSec question ? – Thorsten S. Jan 28 '15 at 18:07
  • You're comparing containers and not plaintext right? – RoraΖ Jan 28 '15 at 18:07
  • 2
    This is definitely more of a programming question. It's troubleshooting a problem, not really asking about any security implications. That said, when you're home-growing solutions to hack together things that weren't designed to integrate with each other in the first place, don't be surprised when stuff breaks in weird ways. What do you really have that's of critical confidentiality which takes up several GB, and for which you're obviously not looking into existing business or enterprise-class solutions, to begin with? – Iszi Jan 28 '15 at 18:32
  • 1
    Possibly Windows started an automatic deframentation of your mounted truecrypt container, rearranging the data. – Michael Karcher Jan 31 '15 at 23:58

I think you're making a large mistake in your approach.

Modern encryption algorithms work on blocks of data. AES encrypts/decrypts 16 bytes at a time. Normally Cipher Block Chaining (CBC) mode is used. In this case the previous cipher is mixed with the next block of plaintext before the encryption occurs. An initialization vector (typically 16 bytes of random) is used as the initial "ciphertext" to start everything off.

In this scheme a change in a single byte of plaintext should change the entire block's output. You should see a significant change in the data. So it's actually odd that there was only a 5% change in the data previously.

The Bad News
I have an feeling that your data is corrupted, and I'm not sure if this ever actually worked (based on your initial description). With CBC anyway, you cannot simply write out the bytes that have changed. This would corrupt future blocks of decryption since previous blocks of cipher rely on the resulting plaintext to decrypt. As described above, changing any portion of a block of cipher would then significantly change the output of plaintext. Resulting in corruption.

Unless I'm misunderstanding what your script is doing.

  • Writing only bytes which were changed (if it's like a patch) makes two identical copies so no need to worry. – Vlad Apr 28 '16 at 22:57

Not the answer you're looking for? Browse other questions tagged or ask your own question.