Ok, I've found the answer. AV companies use clustering/classification techniques in their propietary analysis tools. These are some examples:
Kaspersky uses Astraea : a machine-learning based malware analysis system. Astraea automatically analyzes notifications from protected computers and helps uncover previously unknown threats. By using the threats’ metadata (like age, origin, filename, file path and more) the system is able to fully detect threats without information about the file contents (only static features?).
Comodo uses Valkyrie : it's a cloud based file analysis platform providing static, dynamic and even human expert analysis for submitted known and unknown files. Valkyrie processes over 200 million unknown file submissions each day. Some of the Machine Learning techniques Valkyrie employs include support vector machines, naive bayes, decision trees and random forest classifiers. Additionally, Valkyrie will employ linear discriminant analysis, stochastic gradient descents, hidden markov models and neural networks, just to name a few.
Avast uses Medusa : a GPU-powered classification system. Every Medusa node uses two or four Nvidia GPUs. Each sample is represented by a constant-sized feature vector consisting of approximately 100 attributes. Taking into account the nature of the attributes, they ended up with several distance operators and a weighting scheme that equalizes the importance of the attributes. They use a kNN classifier. Their internal systems handle around 250,000 new PE files every day. They claimed to use complete linkage, this means their algorithm to find the clusters could be the agglomerative hierarchical algorithm.
Avira’s Cloud Protection : Avira uses big data analysis to automatically determine if a newly discovered sample belongs to a known family of malware. At any given minute, they are processing 1.3 Million malware updates on their servers worldwide. In one day alone, their Avira Protection Lab honeypots will receive more than 130,000 new malware samples to analyze.
Microsoft Azure Machine Learning : They use probabilistic machine learning to find patterns in the data that may have been undetected by the more blunt deterministic technique. The algorithm runs without human intervention and produces a map of sorts. The map shows populous clusters, less populous clusters, and perhaps a few outliers or very sparsely populated clusters. At this point, a human expert is called in to look at the clusters and decide if a cluster is a normal process or malware.
BitDefender : uses several machine learning techniques (Perceptrons, Neural Networks, Centroids, Binary Decision Tree, Deep Learning, etc.) for several tasks: filtering URLs, identifying malicious files and to classify emails.
Symantec uses Cynic 
References (I haven't enough reputation to put them as links):