15

I download all kinds of stuff from shady places. But I want to be sure there are no nasty viruses in these files. Is it enough if I just check the hash?

For example, when I go https://www.virustotal.com here and upload my file, I think it computes the hash and looks for it in the database. If the hash is clear - the file is clear. Is that true?

Can I add a virus and modify the file so the hash would be identical to the clean file?

EDIT: The main goal of my question was this:

Suppose we have a perfect AV(find every virus) that has sha256 database, and everyone can upload any file to it.

This AV scans the uploaded file and stores the hash.

Suppose, the client downloaded some file somewhere and calculates its sha256 and searches the hash database.

The database says that there is such a hash in it, and file with that hash has no virus.

What are the chances that client's file has a virus in it?

So this is more like a hashing problem

0

9 Answers 9

10

If the hash is clear on VirusTotal, then that file has been analysed by VT's AV engines and nothing was detected. That's slightly different from not having a virus.

Can you modify a file (by adding code) and come out with the same hash? The concept you are looking for is "hash collisions". Yes, for some hash algorithms, this is possible, but not for the ones that VT uses. It uses MD5 as a possible hash, but is also uses SHA1 and SHA256. It is more possible to create a collision with MD5, but good luck trying to find one, and then whatever you do will not escape SHA1 or SHA256.

13
  • 9
    Finding an MD5 collision is actually extremely easy.
    – forest
    Commented Nov 6, 2022 at 0:24
  • 5
    @forest: Finding a pair of files with the same MD5 hash is easy. If someone wanted to create a pair of programs--one good and one evil with the same hash, and could get someone to vouch for the hash of the "good" one, that person would unknowingly also be vouching for the hash of the bad one. On the other hand, if the file submitted for hashing was not deliberately contrived to share a hash with an evil program, it would be intractiible even using MD5 for someone to make an evil program whose hash would match it.
    – supercat
    Commented Nov 6, 2022 at 6:41
  • @forest you mean it is extremely easy to add code to an existing file and have it come out to the same hash?
    – schroeder
    Commented Nov 6, 2022 at 10:05
  • 1
    @schroeder, sorry for not being clearer. In the attack that I described above, the attackers were able to create a new, fake certificate with the same MD5 hash as an already existing legitimate certificate. See the link above, and note where it reads: the attacker can then apply the collision algorithm documented by Sotirov et. al to create a forged certificate that removes the... and still matches the MD5 hash of the legitimate certificate signed by the CA. There is also more info about this on Wikipedia at en.wikipedia.org/wiki/Flame_%28malware%29#Operation.
    – mti2935
    Commented Nov 7, 2022 at 19:07
  • 1
    @schroeder Probably not easy by most interpretations of the word 'easy'. According to documents.epfl.ch/users/l/le/lenstra/public/papers/lat.pdf, this type of chosen-prefix collision requires ~2^39 MD5 computations to construct. So, doing some back of the envelope calculations, if the attackers had a few thousand dollars worth of GPU's, they might have been able to find this collision in a few days. crypto.stackexchange.com/questions/44151/… is also informative.
    – mti2935
    Commented Nov 8, 2022 at 15:21
41

It depends. If you download a file and verify the hash against a known trusted source, then that's usually safe. For example, if you download a Windows CD and the hash matches the one published by Microsoft, then you can verify the file is the one published by Microsoft, assuming you use a secure hash like SHA-256.

If you trust Microsoft not to produce products with malware, but for whatever reason downloading directly from them isn't possible, then this is a good way to ensure your software is free of malware. However, all a secure hash like SHA-256 tells you is that the software isn't modified (and an insecure hash like MD5 or SHA-1 tells you nothing at all). If you got the hash from Joe Q. Public's Warez Emporium, then you know the software you got is unmodified from what was uploaded, but it doesn't tell you any more than that, because we can't trust that site to ship only non-mallcious software.

Some sites like VirusTotal may allow you to search by a hash and see what certain antivirus software said for software at the time that the file was uploaded. They might have later determined that the software was malicious, or it might be malicious but neither match a known threat nor the heuristics.

That's because the decision about whether software is malicious is a human judgment about its behaviour given the totality of the circumstances. If I'm using Google Chrome with its password manager and it encrypts my passwords and sends them to Google, that's probably okay. But if it encrypts them and sends them to you, it's not, because you're not the trusted third party I had intended to entrust with my passwords. It's fundamentally impossible for software to make this assessment, and all it can do is look for patterns of software that match previous malware.

However, in the ideal situation with a perfect antivirus, assuming you're using SHA-256 or another secure hash like a SHA-2, SHA-3, or BLAKE2 hash, then it's functionally impossible to find two files with the same hash, so if your perfect antivirus says a file with that hash is free of viruses, then verifying that your file has that hash is sufficient to prove that it's free of viruses, because they must be the same file. As I said above, if you use MD5, SHA-1, or an insecure hash, then you can't make that assumption, and you don't know anything interesting by using such a hash.

10
  • 27
    Hashing ensures that the software has not been tampered with from source to you, but if the source is bad, then the hash doesn't protect against that. If I deliberately wrote a piece of malware, and created a hash for it and then uploaded it to whoever, that hash will simply say that nobody else modified it. It doesn't say the software is benign.
    – Nelson
    Commented Nov 6, 2022 at 12:26
  • 13
    It's like using https to connect to www.evilhackers.com. You can be sure that nobody else can see what's going on. And that you are connecting to the genuine www.evilhackers.com sit. But that site is run by evil hackers...
    – gnasher729
    Commented Nov 7, 2022 at 0:44
  • 1
    Great answer, but I think it could be improved by also more directly answering the question "Can I add a virus and modify the file so the hash would be identical to the clean file?" Which is theoretically yes, practically no (assuming virustotal uses a secure hash).
    – RHawkeyed
    Commented Nov 7, 2022 at 10:17
  • 2
    This answer seems to be talking about a different hash than the one the OP is. He's not talking about the hash provided by the publisher, but a hash maintained independently by the AV site.
    – Barmar
    Commented Nov 7, 2022 at 16:45
  • 2
    @gnasher729 I would assume evil hackers would use the RFC 3514 evil bit protocol so you know which of their packets are evil or not.
    – corsiKa
    Commented Nov 8, 2022 at 5:43
11

The hash only shows you if the file was corrupted or altered, not if the file is clean. Someone can send you ransomware with a hash, you calculate the hash and it will show the file wasn't modified, but the file is malicious.

Virus Total will show if the file can be seem as malicious depending on a few factors, but it's trivial to create malware that shows as clean there, but it's not. Ask your favorite search engine about Fully Undetectable Malware and read a little.

Anti Virus solutions are like your immune system: they protect you against threats they know, and threats that look like the things they know. Newer ones infect your body. So newer malware that does not look like older malware can infect you, and the hash does not have anything to protect you.

4
  • 4
    According to AV vendors, roughly 25% of all malware is detected. This number (plus minus a few percent) is stated by the vast majority of major AV vendors, so it's safe to say the truth is likely in that ballpark. Commented Nov 6, 2022 at 2:58
  • 1
    To push the immune system comparison further: anti-virus programs are like the adaptive immune system, which reacts to known threats. Humans also have innate immune system that reacts to damage done by any infection - closest parallel in computers would be a system administrator who notices that something is not working right.
    – jpa
    Commented Nov 6, 2022 at 7:48
  • Virus Total will show if the file can be seem as malicious depending on a few factors, but it's trivial to create malware that shows as clean there, but it's not. Can you provide an example?
    – Bert
    Commented Nov 8, 2022 at 15:59
  • 1
    Search for "Fully Undetectable Malware"... There are several methods for encoding, encrypting and modifying malware to show as clean on Virus Total.
    – ThoriumBR
    Commented Nov 8, 2022 at 16:10
2

The most important thing that I don't see mentioned in the other answers is that a great deal of malware doesn't have a single (or a finite number) of recognizable hashes.

  • There are a great number of malware that is not a single file, but travels by attaching or embedding in another file. Good luck catching it with a hash only.

  • There is such a thing as a polymorphic malware. Its components and padding content are shuffled every now and then so not only hash search is useless, but pattern matching is hard as well.

  • Finally, there is such a thing as still unknown malware. Creating a mediocre, but pretty much functional malware is easy and a lot of people do this for both fun and profit. Malware detection inevitably lags in this regard.

2

But I want to be sure there are no nasty viruses in these files. Is it enough if I just check the hash?

No, cryptographically secure hashes (aka message digests) provide integrity guarantees. Which means that you get a file (e.g. document, program etc) from a trusted source along with its hash that was produced and is provided by the trusted source, you calculate the hash of the file yourself, compare it with the provided hash and verify whether the file was altered after it left the trusted source and before it reached you.

Integrity, however, does not mean that you know what the program does. So, you cannot tell whether a program contains a virus by checking its integrity, if you don't have a digest of the original program available to compare with (in case the program is a virus then signature based detection may apply - see below)

For example, when I go https://www.virustotal.com here and upload my file, I think it computes the hash and looks for it in the database. If the hash is clear - the file is clear. Is that true?

Your hypothesis implies that VirusTotal has already hashed the original version of your file and has its hash stored somewhere for it to be compared with uploaded programs. However, this is not the job of antiviruses but of file integrity checkers. The job of an antivirus program is to analyze files in order to identify whether the file contains malware and probably act on it. Because a file may contain malware in many forms (e.g. polymorphic viruses) the identification cannot be based solely on hash checking (e.g. be signature based) but rather on more complicated methods, with most notable being behaviour analysis. Nevertheless, a virus can evade detection by using special approaches, which makes detection hard to achieve. This means that you cannot be sure that if a virus is not detected in a file the file is not actually infected. As such, the general answer to your question is no.

Can I add a virus and modify the file so the hash would be identical to the clean file?

In principle yes. The same hash can be produced by two different blocks of data, although in cryptography it's really difficult to do it; cryprographically secure hashing algorithms are designed in such a way as to make finding collisions very hard.

EDIT:

The edit in the question changes the context. The assumption is that you have an AV that can detect every virus out there. Aside from the fact that such an AV does not exist, it does not need to have a database with hashes from programs; since it can detect any virus, you can just upload any file and it will be able to say whether the file is infected or not.

So, having a database with hashes may provide some speed gain but you can't have a database with hashes of all available programs in the world (the space requirements would be probably unacceptable). This is especially true since files can be infected by polymorphic malware (see my answer above).

5
  • 1
    Finally someone actually correctly answered the question and pointed out the difference between file integrity / hash checking and virus scanning.
    – NotThatGuy
    Commented Nov 7, 2022 at 1:46
  • It looks like you have completely missed the point of the question. The whole point is to create a hash of the file and check on VT. Not just comparing hash from the developer. The hash in this use case is not about integrity.
    – schroeder
    Commented Nov 8, 2022 at 12:09
  • 1
    @schroeder It looks like you have completely missed the point of my answer. The whole point is to explain how hashes can be used to verify that a file is not altered, how hashes may be used by VT and why this is not adequate for what the OP is looking for.
    – user284677
    Commented Nov 8, 2022 at 12:14
  • Take a look at the edits to the question. It clarifies the inherent context.
    – schroeder
    Commented Nov 8, 2022 at 12:16
  • @schroeder I would say that the clarification changes the context of the question, rather than simply clarifying it. But I've expanded the answer to account for the changes.
    – user284677
    Commented Nov 8, 2022 at 12:39
0

Assuming that the hash algorithm hasn't been broken yet, if two files have the same hash there's an extremely high probability (99.999999999999999999999999...% with about 75 9's) that they are the same file. So high, that we just assume they are.

There's no known way to add a virus to a file but keep the same SHA256 hash. Note that older algorithms such as MD5 and SHA-1 have been broken - that doesn't mean we know how to add a virus and keep the same hash, but it does mean we know how to do things that aren't supposed to be possible with any hash. So don't use those.

So if you put the SHA256 hash into Virustotal and it finds a match, that means it already scanned that exact same file. If it says no viruses detected, it means no viruses detected, in that exact same file that you downloaded.

It does not prove the file has no viruses, only that Virustotal didn't detect any. But that's always how it works. No virus scanner detects 100% of viruses.

0

Not 100%.

In practice, I would trust a download with a known good hash. That's how BitTorrent's peers share files without trackers, and it's also how Windows Update and Smartscreen verify the integrity of the files downloaded.

But in theory there are dangers:

  • The executable might behave differently based on external factors (file name, locale, date, response from third-party server).
  • A file can have multiple different contents (NTFS streams). It's technically possible for other programs to load Stream #2, while you hashed Stream #1.
  • There's a gap between the time you check the hash and the time you execute the file. The safe file might be replaced or modified in between (e.g. the server holds back part of the download for some time, and you hash only part of the file).
  • Not all hashes are created equal. MD5 collisions are easy to make, SHA1 is known to be unsafe, and using a safe hash but truncating it to <128 bits is unsafe.
  • You might have been tricked into visiting the wrong hash database.
  • The hash you selected and copied might not be the hash that was put in your clipboard.
  • Virus-free and safe are not equivalent. A well-meaning program can still wreak havoc, especially if run in an unexpected environment. For example if you downloaded the executable and assets but not the installer that ensures all dependencies are present.
  • If the hash checks are automated, it has its own can of worms, for example what to do in case the hash database is unavailable.
  • Your assumption of a 100% reliable virus checker is not only unrealistic, it's impossible. I might consider adware a virus, or a computer science project might behave strange for learning purposes. The safety of a binary depends on the context it'll run, which is independent of the hash.
-1

It makes you sure that file is not manipulated or changed. But,

  • it may be manipulated at future
  • software is malfunctioning
  • software loads another modules or calls web services that they could be infected
5
  • 2
    1. If it is manipulated in the future, tampered with - the hash changes. Malfunction/bugs are not part of the question as far as I get. Third point in my opinion is valid. Commented Nov 6, 2022 at 16:48
  • The goal of file integrity / hash checking and virus scanning is to check whether the file is fine right now, so the problem of future manipulation doesn't really apply (if something on your computer manipulates it, it means your computer is already infected, which is the thing you were trying to prevent by doing the check in the first place). And software that is malfunctioning or that loads another module would need to have some modifications to its files, which can be detected (somewhat reliably) through file integrity checking and virus scanning.
    – NotThatGuy
    Commented Nov 7, 2022 at 1:30
  • @NotThatGuy in the first part of your sentence the "virus scanning" is misplaced. If the file doesn't change the behavior imbued in it won't change. However, consider certain timebomb-type malware (e.g. Michelangelo of the past which "went off" only under certain conditions or on a particular day. So the judgment passed now on the file may be subject to change without the file ever changing. And so the "fine right now" is a verdict with a potential expiry date. Commented Nov 8, 2022 at 12:23
  • @0xC0000022L You seem to be equating a program doing something malicious and a program being infected.
    – NotThatGuy
    Commented Nov 8, 2022 at 13:15
  • @NotThatGuy I am not. What part of my comment gives you that impression? I worked long enough in the industry to know and appreciate the difference. Commented Nov 8, 2022 at 14:03
-1

I download all kinds of stuff from shady places. But I want to be sure there are no nasty viruses in these files. Is it enough if I just check the hash?

No.

For example, when I go https://www.virustotal.com here and upload my file, I think it computes the hash and looks for it in the database. If the hash is clear - the file is clear. Is that true?

No.

For starters the inverse is also not true. Just because some AV engine claims the file is malicious (or even just suspicious) that doesn't have to be the case. Most laypeople won't even know - let alone care about - the difference between outright malicious (malware) and PUA/PUS (potentially unwanted application/software). And if they did, they would have to decipher a very heterogeneous bunch of detection names across any number of AV engines.

And if they bothered to do that they'd potentially figure out that some of the products listed on VirusTotal share the same AV engine under the hood. And if a single engine shows a false positive but it appears, say, in three products out of four overall, that still makes "only" two false positives.

Last but not least an AV can only find or not find something. VirusTotal calls the latter "Undetected". Undetected is not the same as clean (or good or not malicious).

Can I add a virus and modify the file so the hash would be identical to the clean file?

Theoretically yes, but it's an involved process and feasibility depends on the hash algorithm in question (e.g. if you rely on MD5 then you have relied on the wrong digest algorithm in the last ~17 years).

For all practical purposes this can currently be considered impossible for SHA-256 (your example) in particular, though.

EDIT: The main goal of my question was this:

Suppose we have a perfect AV(find every virus) that has sha256 database, and everyone can upload any file to it.

There is no such thing (perfect AV). I worked almost fifteen years in the industry and can tell you it simply doesn't exist.

And to stress a point I made elsewhere: an AV that claims your system is clean must be using extraterrestrial technology way ahead of anything we know on this planet or it's using a whitelisting approach (which you can get cheaper with AppLocker/SRPs) that would prevent anything unknown by default or it's telling a lie (the most likely scenario).

What a conventional AV can claim is that it hasn't found anything. And there is a big difference between a clean system and one where no (known malicious) entities were found.

This AV scans the uploaded file and stores the hash.

If it scans the file what is it doing? A hash-based (whitelisting) AV would only have to have the capability to prevent files not matching hashes from its own database to run. There wouldn't have to be any heuristics, behavioral analysis (which on the endpoint is usually very limited for performance reasons), any signatures/fingerprints.

Suppose, the client downloaded some file somewhere and calculates its sha256 and searches the hash database.

The database says that there is such a hash in it, and file with that hash has no virus.

What are the chances that client's file has a virus in it?

The chances are mathematically very slim. But we've seen hash collisions during my tenure in the AV industry which is why we never relied on a single hash algorithm for identification (which is what you want).

So this is more like a hashing problem

Yes it is, but I think there is also a misunderstanding of the capabilities anti-malware can offer.

Similarly signing (or specifically code-signing) provide no safety from running malicious code. They attest that the signer signed a file -- and usually when it was signed. You'll have to trust the signer and the certification authorities (CAs) in the certificate chain. And if that trust gets abused there is potentially legal recourse if the CA didn't properly verify the signer prior to issuing the certificate. Signing "just" raises the bar quite a bit. But it's not a 100% bulletproof guarantee that an executable whose signature checks out doesn't turn out to be malicious.


Long story short: if you had different (unrelated) hash algorithms and computed the hash over a the same file using different of these algorithms, each added (unrelated) algorithm should significantly decrease the chances that you ever run into a hash collision. And consequently the chances of it being an entirely unrelated file would shrink to close to zero.

But even if someone stored multiple such hashes of every known file, someone would have to pass judgment on that file first. And that is the weak point in your whole scheme here. There are no perfect AVs. And the load of samples flowing in daily is incredible. Manual analysis is left only to high-profile or otherwise interesting malware and the rest gets usually the sandbox treatment (behavioral analysis) and a number of heuristics then decide into which category a file supposedly falls.

False positives and false negatives exist. None of this is as perfect as in your thought experiment 😉

NB: I used AV (short for anti-virus) because laypeople use the term "virus" for malware in general. However, the term virus only refers only to a subset of the existing malware types.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .