Neural networks, with their ability to learn behavioural patterns from arbitrary data, seem like a natural way to deal with intrusion detection. There are many academic papers on the topic which report good performance and an even better potential.

The question is, are there any real-life implementations? Is there a single intelligent firewall, or a firewall module, or some other sort of an intelligent intrusion detector that actually uses NNs?

  • The closest thing I found is this: roberto.perdisci.com/projects/mcpad, but it seems to be an academic project - I can't tell how usable the thing is. Got to try it out. And yet... I like the idea of getting AI involved. Commented Aug 30, 2012 at 16:46
  • NN is not good way in modelling. It's a very good tool for kids to play with, but for a serious solution you need different methods, which are using MULTIPLE methods, and not just NN. It's like shortcut, has potential in the idea of modelling, but the output is useless, as NN is far too limited in it's ability due to the fact, that to model behaviour you would need to use at leat 8x NN and connect them, which is not NN any more, because not all parts have to be, and cannot be processed as NN. Commented Aug 30, 2012 at 19:08
  • It's like you would be inspecting a cable as telephony solution. Cable is not enough. You can drag cable and therefore send a message, but it's not a real, usable solution. If there is anything with NN, it's just marketing bauble, and probably not good. Commented Aug 30, 2012 at 19:08
  • 1
    You're not being objective, Andrew. Obviously, a NN with 3 inputs and an output is just as likely to "work" as a single neuron is likely to approximate the XOR function. However, the field of NN research is vast and versatile, and - surprise! - it includes a thing called NN ensemble (I am now referring to your idea of interconnecting NNs). I know there is no out-of-the-box solution, for the mere reason that there never is an out-of-the-box solution, whenever the problem to be solved is harder than multiplying 2 by 2. But I am a researcher. I am looking for problems which haven't been solved :) Commented Aug 30, 2012 at 19:28

5 Answers 5


There has been an enormous amount of research into using machine learning techniques for anomaly detection, i.e., to scan network traffic and detect intrusions. However, this research has had very little practical impact. These techniques have seen little deployment and are rarely used in practice.

Why not? There are a number of reasons.

First, these systems tend to have a high false alarm rate. They often raise multiple spurious alarms per day (sometimes even dozens per day), which takes up system administrators' time. This is a fundamental challenge for anomaly detection systems, because they suffer from the "needle in a haystack" problem: billions of packets traverse your network every day, and almost all of them are benign. If the algorithm has a false alarm rate as low as 0.1%, that's still thousands of packets being spuriously flagged. To be practical, the anomaly detection algorithm needs to have an exceptionally low false alarm rate, which is very challenging to do well -- for the same reason that it is very difficult to detect terrorists in airport screening, without introducing a lot of false alarms that cause everyday folks to have to be searched.

Second, anomaly detection systems tend to be not very robust. They focus on detect unusual or novel patterns in your network traffic: anything out of the ordinary. The consequence is that, any time something changes about your network, no matter how benign, they tend to raise alarms. Did your website just get slashdotted? Blam, spurious alarms go crazy. Did some user install a new application that plays novel NAT traversal games? Blam, here come the spurious alarms. Did someone just install IPv6 for the first time? Blam. Someone connect a new mobile phone with a wonky TCP/IP stack, that sends out broken packets? Blam. You get the idea.

If you want to read more about the challenges of innovation in this area, I would recommend the following research papers:

  • Thanks a lot! The first paper looks like an answer I was looking for. Commented Sep 3, 2012 at 8:27
  • The only successful anomoly based IDS Ive heard of in production is fireeye. I'm extremely skeptical but so far Ive heard good things
    – user11869
    Commented Oct 3, 2012 at 11:01

The problem with ML is with the training. Over-training leads to matching the exact training set, making the learning non-generic.

Since my current employer develops scientific computing libraries for Python, I can point you to map-reduce on Disco for finding common clusters in log files: https://github.com/JensRantil/disco-slct

That's not a NN, but it is a way to analyze the data.

With the help of SLCT, one can quickly build a model of logfile(s), and also identify rare lines that do not fit the model (and are possibly anomalous).

  • Thanks! I do my humble bit of research in ML, I know its limitations ;) I guess it will be correct to assume the intrusion detection problem hasn't been solved with ML techniques yet. Interesting. Commented Aug 30, 2012 at 16:56
  • Ha. Sorry. Just read your profile. I know very little about ML, but I hope my link helps. Commented Aug 30, 2012 at 17:00
  • @rox0r: Over-training in neural networks can lead to so-called "high variance", which is what you describe: overfitting the training data. But it's fixable. If you use something called regularization, you can avoid most of the pitfalls of overtraining and you can sometimes find a suitable solution. Regularization can be thought of as penalizing all of the training examples enough that overfitting is not possible. Commented Aug 16, 2016 at 21:18

I doubt you can find any commercial product since this domain is highly commercialized and their is almost no open source implementation available and most of the work is done in is closed ecosystem. There has been a discussion on this topic you can find at link The only open source tool i found was OSSEC it is a Host-Based Intrusion Detection and there is a recent research on integrating it with AI techniques and there is also a book on it you can find interesting.

  • Awesome! Thanks a lot. I'm just trying to figure out how much of a trodden ground the idea itself is. Apparently, the idea is old, but a stable, universally accepted solution is still pending. Now, taking into account the recent breakthrough in.... Wait, I shouldn't have said that :P Commented Aug 30, 2012 at 18:41

As AI is my primary job, I can tell you straight away that the NN alone is useless solution for dynamic architecture.

What you are looking for, is the automation between NN <=> Operator.

This works the following way, for example:

  • NN is trained to recognize existing malware
  • New malware is inspected by response team.

There are no existing applications like you say because it's not done this way, it doesnt have future and it's inherently bad design to use NN to make dynamic architectures.

What is used is actually Bayesian classification as well other heuristic methods. Example implementation is Cloud Exchange by microsoft. It is also using multi-layer architecture.

If you really want to try hard and go the scientific way, you need to analyse each security layer in real-time and make the workflow with the operator.

This forum is not either good place to ask such questions, as there are no experts in AI, or even not engineers, you can try actually MSDN which is better place.

  • 2
    Thanks for your answer. I asked the question on this forum because I wanted answers from people proficient in security (not in AI). I think I got what I wanted. The rest is up to me :) Commented Aug 30, 2012 at 19:34

Found yet another paper on the topic: "Anomaly-based network intrusion detection: Techniques, systems and challenges", by García-Teodoro et al, 2008.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .