A packer is a way of obfuscating an executable program, i.e., transforming so the result is still executable and has the same effect when run, but looks different (so it won't be detected by static anti-virus). Bad guys often use custom packers to obfuscate their malware, to make it less likely that anti-virus will detect the malware or to make it harder for anti-virus vendors to reverse engineer the malware and figure out what it is doing.
Is it possible to detect whether a particular executable has been packed with a custom packer? In other words, given an executable, I'd like to classify it as either "has been packed with a custom packer" or "hasn't". Are existing anti-virus tools good at detecting whether an executable has been packed with a custom packer?
I'm especially curious about the following easier variant of the problem. Suppose I know about a particular custom packer that's being used by bad guys. Is it feasible to recognize executables that have been packed with this packer? In other words, given an executable E and a packer P, I'd like to classify E as either "was packed by P" or "wasn't". Are there known techniques for doing so? How effective are they?