Tell me more ×
IT Security Stack Exchange is a question and answer site for IT security professionals. It's 100% free, no registration required.
up vote 4 down vote favorite
1

I'm using Snort in a lab environmnet with artificially generated traffic.

I'm looking to build a confusion matrix from known generated attacks.

True Positives and False Positives are easy to parse out of the alerts file, but I'm looking for a way to quantify True/False Negatives.

These would be instances in which Snort could have alerted, but didn't.

I'm not sure how closely related this is to number of packets, because I'm not sure if Snort could potentially alert multiple times per packet.

Any help is appreciated.

share|improve this question
Might be worth migrating this to security.SE. – Ladadadada Nov 29 '11 at 23:39

migrated from serverfault.com Nov 29 '11 at 23:47

1 Answer

up vote 3 down vote accepted

The key thing you need is ground truth. You need an a priori way to know which parts of the traffic are in fact attacks, and which are not.

Once you have ground truth, then building the confusion matrix is straightforward. For each attack in the traffic, you check whether Snort detected the attack or not (this gives you a count of True Positives and False Negatives). For each non-attack in the traffic, you check whether Snort alerted about an attack or not (this gives you a count of False Positives and True Negatives). Now you put those four numbers into your 2x2 confusion matrix.

If you don't have ground truth, you cannot build the confusion matrix.

There is no way to use Snort to look at negatives in terms of alerts.

share|improve this answer
I do have ground truth since I am the one running the attacks. Unfortunately 1 missed attack is not equivalent to 1 false negative when I'm focusing on Alerts. To help clarify, if I have 10 attacks run on my network, there could be 300 Snort alerts. Of these 300, 250 could be True Positives, while 50 are False Positives, because there is not a one-to-one relationship between alerts and attacks. I'm trying to get this same data for negatives. Thanks for the help. – Evan Dec 2 '11 at 19:02
@Evan, I think you're saying you want the units for "False Positives" to be "number of (bad) alerts". The units for "False Negatives" is "number of (bad) attacks". There is no way to measure false negatives in terms of number of alerts. So you can certainly count the number of false-positive alerts (i.e., alerts that don't correspond to a real attack), and the number of false-negative attacks (i.e., attacks that don't correspond to any alert); they'll just be in different units, so you can't directly compare them. It's just a question of knowing what you want to measure, then measuring it. – D.W. Dec 3 '11 at 4:06
Thanks. That's exactly what I've been doing (using different units for each). Thanks for your comments. I've been using another sensor as well, which worked as a binary classifier, so I was able to have a more specific look at true/false negatives. – Evan Dec 3 '11 at 6:17
Sounds good! One more random comment: an industry person I know once taught me that "a false positive is any alert the operator didn't want". I think that's insightful. For instance, if one attack generates 20 alerts, one might question whether all of those are truly wanted. It is possible that the operator might prefer to see just 1 alert (leaving 1 alert and 19 false positives) -- or that the operator might prefer to see all 20 (leaving 0 false positives). You probably can't tell on your own; you have to work with the users of your software. – D.W. Dec 3 '11 at 6:33

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.