7

I would like to compute the MD5 hash of a file

The file is uploaded to my webserver by my clients The file is uploaded in pieces (called chunks) The client also sends the md5 hash of the file so that the file integrity could be verified

The chunks can come in any random order. (The first chunk doesn't always have to come first) My web server stores the chunks in persistence storage.

Is it possible for me to compute the hash (or any digest) of the individual chunks and arrive at the hash of the file?

Improve this question
6
  • Why do you want to do this?
    – user163495
    Commented Dec 10, 2020 at 10:14
  • (It's a explained in the original description) My application (multiple instance) gets the files chunk by chunk, and I am storing the file as chunks in a storage. I need to verify the integrity by hash
    – Nachiappan Kumarappan
    Commented Dec 10, 2020 at 10:27
  • 1
    oh okay... I wish i don't need to do this. The system is implemented already and now we pulling all chunks and we are computing MD5. Planning to change the implementation to it more intelligently without breaking the contact
    – Nachiappan Kumarappan
    Commented Dec 10, 2020 at 11:03
  • 1
    Might be worth a mention that if you are using HTTPS then integrity checking is not really necessary at all - HTTPS already has that built in.
    – Conor Mancone
    Commented Dec 10, 2020 at 18:22
  • 1
    @ConorMancone: HTTPS only prevents corruption of data in transit. Hashes, used properly, can also guard against other problems such as having a file change between the time one group of chunks is requested and the time another groups of chunks is requested.
    – supercat
    Commented Dec 10, 2020 at 20:49

1 Answer 1

Reset to default
13

The classical solution is using the Merkle-Tree. To use the Merkle-Tree for uploading, the client forms the Merkle-Tree and calculates the hashes, and sends you the root hash in advance in a secure channel with a digital signature.

In the Merkle Tree, the data (chunk) can arrive at the server in random order with the sibling hashes of the path from the data to the root. The server can use the root hash and the data and sibling hashes to check that the data is valid.

This will increase the bandwidth with O(log n) where n is the number of parts. With MD5 it will be 128*log n ( The log n is used for the height of the tree).

For example for a 1GB file, with each part is 1MB and using a 32B Hash function, like SHA256, for this the client needs to store 64KB hash values and for sending a 1MB chunk with a 320KB hash data will be transmitted and the overhead of the transfer will be 0.031%

The cost of verifying for each chunk will be O(log n), and remember the Cryptographic hash functions are very fast.

and don't use MD5 it is no longer considered to be secure Cryptographic hash function, rather prefer at least SHA256 or use the Blake2 which has speed records. There are also parallel hashes like ParallelHash of SHA3 and the Blake3. This can fasten the calculations whenever the parallelization provides benefits.

For a nice and comprehensive guide to how the Merkle-Tree works see Squeamish Ossifrage' answer on Cryptography.SE

Improve this answer
11
  • 4
    While it is accurate that MD5 is no longer secure, that doesn't mean MD5 is useless for the purpose of file verification or other non-secure applications. When your purpose is trying to verify the transmitted data between two trusted parties, there's nothing wrong with using MD5.
    – Nick2253
    Commented Dec 10, 2020 at 16:39
  • 2
    @Nick2253, if you don't trust the connection between those two trusted parties, and you need to worry about someone in the middle trying to engineer a collision (to let them replace the document without the hash appearing to change)... that assertion is problematic.
    – Charles Duffy
    Commented Dec 10, 2020 at 16:56
  • 2
    @Nick2253, ...indeed, I'm curious about exactly which kind of attack profile it is for which you're asserting that md5 is still a suitable defense.
    – Charles Duffy
    Commented Dec 10, 2020 at 17:00
  • 2
    @Nick2253 the security need is defined according to your threat. On the Cryptographic side, we usually consider the adversary capabilities only computationally bounded with any technique they can apply old or new. I never suggest using MD5, collision easy, pre-image is faster than brute-force. While there are secure alternatives faster than MD5 like the Blake2b why use MD5 at all? What happened to the active adversary that can replace the MD5 hash of a package with a secondary image? If you only consider some errors on the path, then an error-correcting code can be faster than all.
    – kelalaka
    Commented Dec 10, 2020 at 17:08
  • 2
    @CharlesDuffy, MD5, as far as anyone knows, is still preimage resistant. It's completely broken for situations where the attacker can freely choose every file being hashed (such as when getting a third party to sign a file), but in situations where at least one file is not under the attacker's control, there are no known attacks.
    – Mark
    Commented Dec 10, 2020 at 22:27

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .