What are the technical implementation differences between signature-based scanning and heuristic-based scanning?

Question

Signature-based scanning works by comparing something to a database. So does this basically amount to scanning a file x for string foobar?

I know heuristic-based scanning can detect new threats that aren't in a database yet. How does it do this? Does it use regular expressions, for example scans files for foo*bar where * is a regular expression operator?

I've also heard that heuristic-based scanning looks at the behavior of a file, but this sounds like something entirely different such as emulation or sandboxing.

Also how do heuristics apply to IDS and Firewalls canning network traffic?

Answer 1

Heuristics - whether used in AV, antimalware, or on a network - work on the premise of "straying from the norm." To avoid a lengthy post, please read "Understanding Heuristics." Now to give a non technical analogy that is applicable to most (networks, AV, etc): Imagine you are a security guard at a building (AV, network appliance, etc) and you are told: red cars are dangerous. (signature) You will be on watchful of red cars. This is a static rule you were given. You would not be useful in many situations. As a heuristic security guard, you would flag a blue car that carried five passengers. But why? Because you learned to look for anomalies, you built baselines.

Applications and network traffic follow patterns, for example: "When someone visits a website, they perform X Y and Z functions." This pattern repeats to where an application deems it normal, any straying from this norm is an anomaly. Your application adjusts: "If everything looks like A, and I already know that B may be bad, I wonder how the intersection, or variation may look like, I need to be wary that anything looking less than a stellar A, may be related to B, and flag it." This is how say IBM's Proventia and others did this. "After so much time, A always equals A, when it doesn't, let us go back historically and see if the addition (B) meets criteria for known/suspicious malicious actions."

Now we are able to flag on the network: "Under normal conditions, a user connects to port X and sends N amount of data. Strangely we now have a user that has connected 10 times in under one minute, and has sent Z amount of data. ALERT" This is also how IDS systems work, and why - when anomalies are detected - most administrators have to begin fine tuning for false positives. On the malware/AV side of the equation, the same applies: "File A.exe is not supposed to rename itself, hide its process, and fire off multiple connections, we have never seen an application do so, therefore we are suspicious of it." AV/Antimalware will compare its known knowns (signatures) against current behavior, and against baselines."

Answer 2

Heuristic scanner (for a file) can look/do for various things, including:

• sandboxing the application and analyse the behaviour (which files are created/deleled/modified, ...)
• looking for hooks to functions (direct disk writing, TCP socket binding, ...)
• strings like filenames (read/write of system files, ...)
• memory residence ability
• decryption of the program at launch
• decompiled machine code analysis

The detection of one or multiple of these elements can point toward a possible threat. A program which is resident and modifies the hosts file, is very likely to be ill-intended.

As for network based analysis, I am not very familiar of it. But I'd say, if there is a machine dedicated to this, it would be possible to run analysis of captured file and/or reported files.

(possible interest in Rethinking Antivirus: Executable Analysis in the Network Cloud)