Heuristics - whether used in AV, antimalware, or on a network - work on the premise of "straying from the norm." To avoid a lengthy post, please read "Understanding Heuristics." Now to give a non technical analogy that is applicable to most (networks, AV, etc): Imagine you are a security guard at a building (AV, network appliance, etc) and you are told: red cars are dangerous. (signature) You will be on watchful of red cars. This is a static rule you were given. You would not be useful in many situations. As a heuristic security guard, you would flag a blue car that carried five passengers. But why? Because you learned to look for anomalies, you built baselines.
Applications and network traffic follow patterns, for example: "When someone visits a website, they perform X Y and Z functions." This pattern repeats to where an application deems it normal, any straying from this norm is an anomaly. Your application adjusts: "If everything looks like A, and I already know that B may be bad, I wonder how the intersection, or variation may look like, I need to be wary that anything looking less than a stellar A, may be related to B, and flag it." This is how say IBM's Proventia and others did this. "After so much time, A always equals A, when it doesn't, let us go back historically and see if the addition (B) meets criteria for known/suspicious malicious actions."
Now we are able to flag on the network: "Under normal conditions, a user connects to port X and sends N amount of data. Strangely we now have a user that has connected 10 times in under one minute, and has sent Z amount of data. ALERT" This is also how IDS systems work, and why - when anomalies are detected - most administrators have to begin fine tuning for false positives. On the malware/AV side of the equation, the same applies: "File A.exe is not supposed to rename itself, hide its process, and fire off multiple connections, we have never seen an application do so, therefore we are suspicious of it." AV/Antimalware will compare its known knowns (signatures) against current behavior, and against baselines."
