Confidentiality of file hashes

Question

Context: I have a directory full of sensitive company files. I want to compare these files with a colleague at a remote location, without the need for a secure connection.

Is it safe for me to publicly publish a list of the SHA-1 hashes of these files?

I understand that it's impossible to reverse the hashing algorithm, but since the hashes are calculated from the original file, is there any chance that an attacker could perform some sort of a brute force attack to rebuild the original file? (Note that collisions in this case don't matter.) It's clearly impossible with a 20 page document, but could small files be vulnerable?

What about if I used a more secure hashing algorithm, like SHA512?

(I'm not very familiar with salting, but I think it wouldn't help in this situation because I don't care if an attacker identifies two files as the same, just if they identify the original contents.)

Answer 1

SHA-1 produces a 160 bit hash. Hashing any file longer than 160 bits (20 octets) will therefore necessarily lose information and makes reconstruction impossible. In theory, that is.

On the other hand, hashing files shorter than 20 octets is highly likely (practically guaranteed) to produce a 1:1 mapping. A 1:1 mapping means that without salt, it is trivial to revert the hash to the original contents with a readily existing rainbow table. Even with a typical non-secret salt, it is very possible to run a brute-force attack on very short files, so if you are concerned about that, you should add a salt that is sufficiently long (say, 128 bits) and keep the salt secret (this is not how you would normally use a salt, but you have a different situation, too). You could further add each file's name to the salt (unless what you are trying to do is deduplication), so different files with identical contents have different hashes.

In practice, although the hash cannot be reversed for files larger than 20 octets, small files (but larger than 20 octets) may still be reversed if the attacker is sufficiently persistent. For example, there exist 65536 files with 22 octets^[1] that have the same SHA-1, and you cannot prove which is the correct one. Or can you?
Unluckily the answer is "yes". Although each single of those 65k different files is an equally valid solution from the point of view of the hash, only one of them (or maybe two) will be something that isn't nonsensical random binary garbage. Which is trivial to identify by using a general purpose compression program (plaintext files are compressable, random garbage isn't). Moreover, if a file's name is known, it is usually relatively easy to check its contents against some magic bytes or a particular structure. The attacker only needs to consider files which have magic bytes that correspond to their type.
Luckily, this attack quickly becomes impractical. There are already 10²⁸ files of length 32 which map to the same hash, and the vast majority of files on every computer is longer than that!

And now here is a surprise: The "more secure" SHA-512 is actually less secure in that respect. Since it ouputs 512 bits, it will do a 1:1 mapping for files up to 64 bytes.

My recommendation would be:

1. If you really don't want to (or can't) use TLS/SSH (you know that rsync will do the whole comparing hashes stuff including SSH connection for you, don't you?), use an encryption container such as e.g. Truecrypt. That will prevent someone else from accessing the hashes even if you put the container on the internet publicly on an untrusted server, or if you send them by email.
   This makes every other consideration obsolete. No need to worry about whether the hashes can be reverted if the attacker doesn't know them!
2. Do not use a hash that is bigger than necessary. The chance of one random hash collision in 10¹⁶ files (that's 10 billion times the number files presently on my desktop computer!) with a 160-bit hash is about 10^-15. For ten thousand files, it's 10^-22. In other words, it won't happen in your lifetime. Revision control sets like e.g. Git rely on the fact that collisions simply don't happen. Bigger hashes do not make anything better in your scenario, but they quite possibly make things worse (for small files).
3. Salt file contents with file names and with a sufficiently long random salt that you keep secret.
4. Consider hashing file names as well, since file names alone may convey important information.
5. Don't transmit file lengths. It doesn't give you an advantage, but it may give one to an attacker.

---

^[1] Actually, if you don't tell the attacker the file's length, there are even a few more: there are also 256 files with 21 octets and one with 20 or fewer.

Answer 2

The security you are thinking of with regards to the hash strength and the security you are talking about with the hashes on the internet are two different things. Hashes like SHA* are designed to work quickly so that files you send and the file you receive can be verified to be the same, however, this makes it easier brute force, due to this speed.

What it all boils down to is bits of entropy: the more bits of entropy the file has, the longer it will take to guess. I would say that if your file is more than a (arbitrary limit) 1kb, it should be reasonably safe from brute-forcing.

However, that without a secure connection, you cannot guarantee that the messages have not been modified in transit.

If you are reasonably sure that your hashes are not going to be modified in transit, this should be a reasonable way to compare the two copies of the file.

Answer 3

Question: Do you want to publish the sensitive file names on the internet?

Your approach above seems to imply that you will be publishing your file names on the internet.

You may not want your file names published on the internet.

brain@brain-laptop:~/Secret Files$ sha256sum *
c988f4a50da6021fc70f618faeb5e27891b5de7162fb395b1dfd5b42f76a8070  Blueprints for Secret Lair Island in the Philippines.docx
78530f114e56ed419950e465f79c33f14bcc91eb63acefcf976ab96cd190915a  Secret Plan to Take Over the World.docx

Answer 4

Under some conditions, your hashes might leak sensitive information.

This depends on whether or not an attacker can guess the content of specific files.

As a trivialized example, let's say your files are memos from your boss which say which employee they are going to fire. They are always simple ASCII text files in the form of The next employee we fire is [Name]. When I would know this and I would have a list of employees, I could calculate the hash sums of all possible files:

The next employee we fire is Alice   9a76503a707ae3b58c3a12324622d45dfbbbd0d6a35e9e539104e285e25b2965
The next employee we fire is Bob     0d42e6bbc62c1a308c85dded89b9f300afe2537539fc068eeb99e00f3154aac5
The next employee we fire is Charlie e3a0ab856e1440eba0d520e0c86f6db22a47675371db80dbccb55a207db9a0dc

When one matches, I know the content of the file.

This attack assumes that file contents are possible to guess. Keep in mind that specialized hardware is capable to make millions of guesses per second. Still, when the files are reasonably complex, like with a prose text containing more than a few sentences, this becomes impossible in reasonable time. But be careful when you distribute non-unique files that way, especially files which an attacker has access to but which you are not supposed to have. Like pirated media files, for example.