Methodologies for test & benchmark of Antivirus Software

Question

I want to evaluate a new antivirus product which obviously hasn't undergone the tests by known test labs. So I want to measure its performance in different aspects and come up with common comparisons and measurements that are available for popular AV products.

My question is that, are there any formal or standard methods, test environments, test data and information available to conduct such a test? Like getting a standard collection of malware samples and testing the new antivirus with that and comparing the results with available AVs or in an inferior case, comparing all possible antiviruses myself with available samples and environment.

Should I use virtual machines and sandox like technologies to see if the AV detects and clears malware samples and then revert back to a clean snapshot or are there easier ways to do it? Any kind of tip and guide will be appreciated.

Answer 1

As for test environments, there are several open source mechanisms that combine given mail server with one or many antivirus engines. You can analyze them and reimplement invoking each engine in your own environment to scan a file.

As for test data, look at http://vxheaven.org/ website for samples.

And yes, you should use virtual machines instead of running anything possibly rogue on your production systems.