Is there any hash structure that can be used for secure download of a file and validation of its contents?

Looks like all hashes like MD5 are made for consistency check and are vulnerable to hash extension attack, so it is possible to create another file with the same MD5 by just appending to the end of it.

I am writing an automated service that downloads data over insecure channels and I want to make sure that MITM can not alter file.

UPDATE: Hash is transferred to the user through secure channel. Right now I am using hash + filesize combination, to avoid hash extension attacks. Is it possible to use just single hash function?

UPDATE 2: I misunderstood what a length extension attack is. Quoting @CodeInChaos:

It does not give you a collision, it merely allows you to compute the hash of a longer file given the hash of a shorter file whose contents you don't know

So it is not possible to just append to file with existing hash to get the same hash. Doing this will mean looking for collisions in a longer file, so the question boils down to classic "how strong is the hash function". On the other hand, it is still interesting why SHA3 is not vulnerable to hash extension, but that's another topic.

  • This question is not clear, you seem to be confusing hashing with encryption. A hash is an irreversible mathematical transformation of data, you can't "decrypt" a hash, so you can't use it to protect confidentiality. – GdD Nov 17 '15 at 9:59
  • @GdD it is not about encryption, it is about validation of download. Hash is received over secure channel. File is not. How to validate that the file is not a forged one? – anatoly techtonik Nov 17 '15 at 10:03
  • 1
    That's the point of using a hash - if the file is different in any way than the original, even by a single bit, then the hash result will be completely different. – GdD Nov 17 '15 at 10:08
  • 1
    If you are asking about collisions then yes, collisions are possible, however the source data for a collision would almost certainly be meaningless. – GdD Nov 17 '15 at 11:27
  • 5
    You misunderstood what a length extension attack is. It does not give you a collision, it merely allows you to compute the hash of a longer file given the hash of a shorter file whose contents you don't know. – CodesInChaos Nov 17 '15 at 11:45

Modern hashes like SHA-256 provide a strong protection against hash collision attacks, that is it is practically impossible to create a different program which has the same hash as the original program - no matter if the size is the same or differs.

This still leaves the possibility that the attacker not only modifies the file but also the accompanying hash, which means you have to find a way to transfer the hash in a secure way to the user.

| improve this answer | |
  • Hashes are transferred securely. Modified the question to mention that. – anatoly techtonik Nov 17 '15 at 11:38

You are overthinking this, and your attack scenario is very unlikely to happen.

First, if you are using a secure channel to send the hash (I imagine you are using SSL/TLS), you should use the same channel to send the file. TLS is very secure if rightly implemented, it's a fire and forget solution, and the administrative overhead is pretty negligible after implementation. It's a little more expensive than plain HTTP, but the benefits more than compensate.

Second, if an attacker is able to intercept the transmission, taint the downloaded file, and redirect the changed file to your client, nothing will stop him from doing the same to the hash. We can say that the file will very likely be bigger than the hash. Intercepting a hash would be trivial.

Third, if the attacker have enough resources to intercept the transmission, download the original file, create a collision on a SHA-256 hash and send the changed file, he will have enough resources to ask you to send the changed file to the receiver with a correct hash and don't tell the receiver. And that would be easier to the attacker, and you don't have any protection against.

Don't worry about intercepted and changed file downloads. Worry about error in transmission. Even a CRC32 is enough to verify a download. If this transmission occurs over HTTP(s), you can use a custom header:

HTTP/1.0 200 OK
Server: SuperHTTPServer/20.1
Date: Sat, 01 Jan 2000 01:23:45 GMT
Content-type: text/html; charset=UTF-8
Content-Length: 12345
X-MD5-Hash: cc0c8d81fc54e0bd077a5678eaadf2ff

The custom client will have to read the reader, get the hash, and calculate the result. You can send both the file and the hash using only one transaction.

By the way, the Hash Length Extension Attack is not what you think. As CodesIsChaos said, the attack consists on getting a hash and a partial string, adding something to it, and recalculating the hash even without knowing the original string. And while is possible to create a collision with MD5, is not trivial, and certainly not practical to create on-fly during a file transmission.

| improve this answer | |
  • Thanks for clarification about Hash Length Extension. Wikipedia holds a very bad article about that with all that maths notation. This is not a question about real-time attack, but about prepared, so if MD5 collision is possible to get in a week or a year, then MD5 won't work. – anatoly techtonik Nov 17 '15 at 15:07

MD5 and SHA256 are both hashing functions that are commonly used to compute the checksum of a file for verification purposes. An MD5 checksum is 128 bits long, whereas a SHA256 checksum is 256 bits long, so an attacker would be less likely to find a collision if you use SHA256.

Another thing you can do to provide a way for your users to verify the authenticity of your file is to digitally sign the file. Of course, this requires that you distribute your public signing key in such a way that your users could verify its authenticity.

| improve this answer | |
  • No, I need to give users a hash and then make sure that he will be able to locate and download the right file using the hash alone. Is that possible? With protection from quantum computers etc. – anatoly techtonik Nov 17 '15 at 11:40
  • 1
    @techtonik That sounds like what BitTorrent Magnet links are doing. – Philipp Nov 17 '15 at 12:25
  • @Philipp exactly. How secure are they? – anatoly techtonik Nov 17 '15 at 14:38
  • You sort of make it sound like finding a collision on sha128 would be easy. Right now that's not the case. – Neil Smithline Nov 17 '15 at 15:55
  • 1
    The whole bitcoin network currently earns 25*330$+fees every 10 mins. Since miners probably wouldn't operate at a loss, that means that a the work the network performs each minute shouldn't be worth more 1000$ or so. (It's a bit more complicated due to the varying bitcoin value, but should be a decent estimation) – CodesInChaos Nov 17 '15 at 19:34

Not the answer you're looking for? Browse other questions tagged or ask your own question.