I recently read a paper on The security of Machine Learning. Are there any instances of attacks against machine learning systems besides spam filters?
Yes, there are recent examples of attackers targetting ML systems in order to later evade detection. The term used for the study of this effect is "Adversarial Machine Learning". There are a number of papers:
One of the techniques discussed is the "boiling frog" technique. Named after the effect that one can boil a live frog without it jumping away to safety if the heat is turned up slowly. Attacks using this method attempt to mistrain the ML system by introducing anomalous stimuli at a very low anomaly factor, but with high frequency. Over time, the anomaly factor increases until the ML system learns to ignore truly anomalous stimuli, at which point the actual attack on the target systems begin. Because the ML systems have been trained to ignore it, the attack goes undetected. (The same technique has been used for years against people and security guards).
This can be done with spam filters, IDS, UBA systems and Netflow analyzers.
For instance: if you have gained the credentials of a valuable user, and you want to be able to log in as that user during a time when that user does not normally log in, you can start logging in using those credentials at times only slightly different from the user's normal pattern while not performing any unusual activity. Once you have seeded the logs with enough logins around the time when you want to perform your malicious activity, you can make your attempt and know that you will not trigger alarms for the anomalous login time. By seeding the logs with non-malicious activity, secondary and tertiary analysis of the activity will support the decision that the activity is not anomalous.
Are there any real instances of attacks against machine learning systems?
Probably the most well known systems based on ML in information security are spam filters. Since sending spam is a lucrative business these filters get attacked a lot by making they look more like non-spam for a machine. Many of the attacks target the feature extraction, for example by including invisible non-spam text into the HTML so that spam filter thinks that the majority of the text is non-spam and classifies the message as such. Other ways to evade the ML based filter is to use alternative spellings for phrases which got learned by the filter as spammy, i.e. use of misspellings, homographic characters, special unicode characters like zero-length spaces etc.
There are other uses of machine learning and heuristics in information security. They are not that well known as spam filters but one can find at least papers which deal with detecting drive-by-download attacks based on URL patterns, derive reputation of hosts based on DNS history and whois records, detect malware C&C communications based on the target URL etc. Attackers bypass such methods by making the URL's look more like typical innocent URL's, compromise hosts with a high reputation to spread their malware or use innocent targets like twitter, blogspot etc for C&C communication.