3

Antivirus companies collect samples from their products or the users can directly submit the samples through forms on their websites. The number of collected samples is very big. How can a group of analysts analyze such a big volume of malwares?

I read several papers made by researchers that describe different techniques. One technique is to cluster similar malwares together and analyze only one 'representative' for each cluster.

What are the methods really used by the AV companies?

  • 3
    This is a very broad question that might be better suited for the statistics SE. This is not so much a security problem as it is a big data analysis problem, that just happens to occur in the field of security. My advice is narrowing it ("how do they extract comprehensive fingerprints etc."). A good start might also be also be this doc: downloads.cloudsecurityalliance.org/initiatives/bdwg/… – J.A.K. Feb 8 '17 at 15:01
  • Something similar is discussed here. Most samples will probably be already identified by a hash and thus won't need to be analyzed at all, those that have slight changes due to polymorphism should be rather easy to identify in a datacenter with adequate computing power. Those high numbers are probably just a publicity stunt to sell those AV licenses. – r41n Feb 8 '17 at 15:12
1

Ok, I've found the answer. AV companies use clustering/classification techniques in their propietary analysis tools. These are some examples:

Kaspersky uses Astraea [1]: a machine-learning based malware analysis system. Astraea automatically analyzes notifications from protected computers and helps uncover previously unknown threats. By using the threats’ metadata (like age, origin, filename, file path and more) the system is able to fully detect threats without information about the file contents (only static features?).

Comodo uses Valkyrie [2]: it's a cloud based file analysis platform providing static, dynamic and even human expert analysis for submitted known and unknown files. Valkyrie processes over 200 million unknown file submissions each day. Some of the Machine Learning techniques Valkyrie employs include support vector machines, naive bayes, decision trees and random forest classifiers. Additionally, Valkyrie will employ linear discriminant analysis, stochastic gradient descents, hidden markov models and neural networks, just to name a few.

Avast uses Medusa [3]: a GPU-powered classification system. Every Medusa node uses two or four Nvidia GPUs. Each sample is represented by a constant-sized feature vector consisting of approximately 100 attributes. Taking into account the nature of the attributes, they ended up with several distance operators and a weighting scheme that equalizes the importance of the attributes. They use a kNN classifier. Their internal systems handle around 250,000 new PE files every day. They claimed to use complete linkage, this means their algorithm to find the clusters could be the agglomerative hierarchical algorithm.

Avira’s Cloud Protection [4]: Avira uses big data analysis to automatically determine if a newly discovered sample belongs to a known family of malware. At any given minute, they are processing 1.3 Million malware updates on their servers worldwide. In one day alone, their Avira Protection Lab honeypots will receive more than 130,000 new malware samples to analyze.

Microsoft Azure Machine Learning [5]: They use probabilistic machine learning to find patterns in the data that may have been undetected by the more blunt deterministic technique. The algorithm runs without human intervention and produces a map of sorts. The map shows populous clusters, less populous clusters, and perhaps a few outliers or very sparsely populated clusters. At this point, a human expert is called in to look at the clusters and decide if a cluster is a normal process or malware.

BitDefender [6]: uses several machine learning techniques (Perceptrons, Neural Networks, Centroids, Binary Decision Tree, Deep Learning, etc.) for several tasks: filtering URLs, identifying malicious files and to classify emails.

Symantec uses Cynic [7]

References (I haven't enough reputation to put them as links):

[1] usa.kaspersky.com/about-us/press-center/press-releases/2016/Kaspersky_Lab_Number_of_the_Year_2016_323000_Pieces_of_Malware_Detected_Daily

[2] enterprise.comodo.com/valkyrie/

[3] avast.com/technology

[4] blog.avira.com/virus-hunters-catch-bad-guys-part-2/

[5] microsoft.com/itshowcase/Article/Content/618/Improving-network-security-using-big-data-and-machine-learning

[6] businessinsights.bitdefender.com/machine-learning-apt-detection

[7] symantec.com/connect/blogs/myth-busting-next-generation-threat-protection

0

How can a group of analysts analyze such a big volume of malwares?

The short answer is that they can't and there is no silver bullet. For instance the first sample of Stuxnet was collected around January 2010, while Stuxnet wasn't discovered until June 2010.

AV companies are really good at collecting samples, but deciding which ones to analyse is a problem that hasn't been completely solved yet. So a relevant question is how to choose which samples to analyse. This is where all the different methods of classifying samples as interesting for analysis comes into play. A lot of them are listed in the answer above.

  • Stuxnet being one of the most sophisticated malwares of its time might be a bad example here. – Tom K. Dec 5 '17 at 14:03
  • I don't see how it is a bad example because it is sophisticated? Malware that can escape automated analysis is fairly sophisticated. The question was how a group of analysts could analyse malware a big volume of malware, and it is a problem that the number of samples per day is higher then the number that can be analysed by a group of analysts per day. So sophisticated malware that has been sampled and stored will not be detected because it is not analysed in time. Stuxnet is an example of that. – Kaffekoppen Dec 5 '17 at 15:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.