0

Context: I have a directory full of sensitive company files. I want to compare these files with a colleague at a remote location, without the need for a secure connection.

Is it safe for me to publicly publish a list of the SHA-1 hashes of these files?

I understand that it's impossible to reverse the hashing algorithm, but since the hashes are calculated from the original file, is there any chance that an attacker could perform some sort of a brute force attack to rebuild the original file? (Note that collisions in this case don't matter.) It's clearly impossible with a 20 page document, but could small files be vulnerable?

What about if I used a more secure hashing algorithm, like SHA512?

(I'm not very familiar with salting, but I think it wouldn't help in this situation because I don't care if an attacker identifies two files as the same, just if they identify the original contents.)

  • You say that hashes are not reversible, but wondering if hashes are reversible? You seem to have the answer you are looking for. – schroeder Aug 13 '14 at 21:35
  • I think the main issue with not adding a key is that someone might be able to identify some of the files if they are the same as externally available files, eg. third-party support documents, clipart, open-source, etc. That might let an attacker guess at what you are doing. If you just append a secret key to each file before digesting, that should eliminate the possibility. – Jeff-Inventor ChromeOS Aug 14 '14 at 18:14
  • Related: Attacks on Convergent Encryption. The weaknesses of convergent encryption and of published hashes are essentially the same. – CodesInChaos Aug 14 '14 at 18:44
  • If can produce a shared secret, you could use HMAC-SHA-2 with a key only known to you and the colleague but not to the attackers. – CodesInChaos Aug 14 '14 at 18:46
1

SHA-1 produces a 160 bit hash. Hashing any file longer than 160 bits (20 octets) will therefore necessarily lose information and makes reconstruction impossible. In theory, that is.

On the other hand, hashing files shorter than 20 octets is highly likely (practically guaranteed) to produce a 1:1 mapping. A 1:1 mapping means that without salt, it is trivial to revert the hash to the original contents with a readily existing rainbow table. Even with a typical non-secret salt, it is very possible to run a brute-force attack on very short files, so if you are concerned about that, you should add a salt that is sufficiently long (say, 128 bits) and keep the salt secret (this is not how you would normally use a salt, but you have a different situation, too). You could further add each file's name to the salt (unless what you are trying to do is deduplication), so different files with identical contents have different hashes.

In practice, although the hash cannot be reversed for files larger than 20 octets, small files (but larger than 20 octets) may still be reversed if the attacker is sufficiently persistent. For example, there exist 65536 files with 22 octets[1] that have the same SHA-1, and you cannot prove which is the correct one. Or can you?
Unluckily the answer is "yes". Although each single of those 65k different files is an equally valid solution from the point of view of the hash, only one of them (or maybe two) will be something that isn't nonsensical random binary garbage. Which is trivial to identify by using a general purpose compression program (plaintext files are compressable, random garbage isn't). Moreover, if a file's name is known, it is usually relatively easy to check its contents against some magic bytes or a particular structure. The attacker only needs to consider files which have magic bytes that correspond to their type.
Luckily, this attack quickly becomes impractical. There are already 1028 files of length 32 which map to the same hash, and the vast majority of files on every computer is longer than that!

And now here is a surprise: The "more secure" SHA-512 is actually less secure in that respect. Since it ouputs 512 bits, it will do a 1:1 mapping for files up to 64 bytes.

My recommendation would be:

  1. If you really don't want to (or can't) use TLS/SSH (you know that rsync will do the whole comparing hashes stuff including SSH connection for you, don't you?), use an encryption container such as e.g. Truecrypt. That will prevent someone else from accessing the hashes even if you put the container on the internet publicly on an untrusted server, or if you send them by email.
    This makes every other consideration obsolete. No need to worry about whether the hashes can be reverted if the attacker doesn't know them!
  2. Do not use a hash that is bigger than necessary. The chance of one random hash collision in 1016 files (that's 10 billion times the number files presently on my desktop computer!) with a 160-bit hash is about 10-15. For ten thousand files, it's 10-22. In other words, it won't happen in your lifetime. Revision control sets like e.g. Git rely on the fact that collisions simply don't happen. Bigger hashes do not make anything better in your scenario, but they quite possibly make things worse (for small files).
  3. Salt file contents with file names and with a sufficiently long random salt that you keep secret.
  4. Consider hashing file names as well, since file names alone may convey important information.
  5. Don't transmit file lengths. It doesn't give you an advantage, but it may give one to an attacker.


[1] Actually, if you don't tell the attacker the file's length, there are even a few more: there are also 256 files with 21 octets and one with 20 or fewer.

  • In theory, could double hashing combat the potential for identifying which of the 65k possible results is the correct one? If you hash the file once, then hash it again, it should be impossible to tell which reversal of the second hash is the correct one, since the first hash should be just as 'noisy' as all the other possible results. – ecapstone Aug 14 '14 at 19:15
  • @ecapstone: Then the attacker needs to perform 2 hashes for each possible file contents, but other than that it's the same thing. If you know that e.g. an input file is 22 bytes long, and you get 20 bytes out, no matter how many times you have hashed them (once, twice, a thousand times), there will be 65536 matches. One of these will be the correct one and you don't know which one -- but you can look at the input and tell which one makes sense. The input is readily available, since you've generated it while brute-forcing for a collision. – Damon Aug 14 '14 at 19:59
  • @ecapstone: You should ask yourself how likely it is that a file which is smaller than, say, 36 bytes (that's 2^128 candidate inputs for each SHA-1 hash!) contains data that is so sensible it must absolutely be kept secret. If you really have such files, you should play no games and use encryption. Otherwise, it's needless to worry about. Nobody will be able to reverse a file with that many possible candidates. – Damon Aug 14 '14 at 20:05
2

The security you are thinking of with regards to the hash strength and the security you are talking about with the hashes on the internet are two different things. Hashes like SHA* are designed to work quickly so that files you send and the file you receive can be verified to be the same, however, this makes it easier brute force, due to this speed.

What it all boils down to is bits of entropy: the more bits of entropy the file has, the longer it will take to guess. I would say that if your file is more than a (arbitrary limit) 1kb, it should be reasonably safe from brute-forcing.

However, that without a secure connection, you cannot guarantee that the messages have not been modified in transit.

If you are reasonably sure that your hashes are not going to be modified in transit, this should be a reasonable way to compare the two copies of the file.

0

Question: Do you want to publish the sensitive file names on the internet?

Your approach above seems to imply that you will be publishing your file names on the internet.

You may not want your file names published on the internet.

brain@brain-laptop:~/Secret Files$ sha256sum *
c988f4a50da6021fc70f618faeb5e27891b5de7162fb395b1dfd5b42f76a8070  Blueprints for Secret Lair Island in the Philippines.docx
78530f114e56ed419950e465f79c33f14bcc91eb63acefcf976ab96cd190915a  Secret Plan to Take Over the World.docx
0

Under some conditions, your hashes might leak sensitive information.

This depends on whether or not an attacker can guess the content of specific files.

As a trivialized example, let's say your files are memos from your boss which say which employee they are going to fire. They are always simple ASCII text files in the form of The next employee we fire is [Name]. When I would know this and I would have a list of employees, I could calculate the hash sums of all possible files:

The next employee we fire is Alice   9a76503a707ae3b58c3a12324622d45dfbbbd0d6a35e9e539104e285e25b2965
The next employee we fire is Bob     0d42e6bbc62c1a308c85dded89b9f300afe2537539fc068eeb99e00f3154aac5
The next employee we fire is Charlie e3a0ab856e1440eba0d520e0c86f6db22a47675371db80dbccb55a207db9a0dc

When one matches, I know the content of the file.

This attack assumes that file contents are possible to guess. Keep in mind that specialized hardware is capable to make millions of guesses per second. Still, when the files are reasonably complex, like with a prose text containing more than a few sentences, this becomes impossible in reasonable time. But be careful when you distribute non-unique files that way, especially files which an attacker has access to but which you are not supposed to have. Like pirated media files, for example.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.