2

There are some formulas for risk-assessing security vulnerabilities, like the DREAD formula (Damage, Reproducibility, Exploitability, Affected users, Discoverability -- rate the vuln in each category on a scale from 1 to 10, then add them all up). Is there any evidence that these work better than just having a security professional look at the vulnerability as a whole and say, "Yeah, on a scale of 1 to 5, I give that a 3"?

Here's why I'm skeptical: The times that a formula is useful is when I'm doing something that is out of my area of expertise, like buying a used car, because I don't know anything about what cars are worth. So I would probably end up using a formula that involved the make and model of the car, the number of miles, and any visible damage.

But in assessing security vulnerabilities, the determination of severity is presumably going to be made by someone who has a good intuition of how bad a vulnerability is. In most cases that I've seen, when two experts independently judged how bad a vulnerability was, they came close to the same answer. And when they didn't agree, they certainly weren't going to resolve the disagreement by looking at the formula -- the person whose assessment was different from the formula, would just say that this is one of those times that the formula doesn't work. So I don't know what is the point of the formula. Formulas are useful when non-experts need to make a decision, but the people making decisions about security vulnerabilities shouldn't be non-experts!

So is there any evidence that the formulas have been useful, compared to the intuitive approach? Pretty open-ended about what counts as "evidence", but I'm thinking for example if a company tried the intuitive approach once, and tried the formula approach once, and found that one worked better than the other.

  • 1
    Not everyone on a team is going to have the same experience. Quantifying a vulnerability is more of a communication tool than anything. If there is mistrust between team members, that will hamper communication regardless of any quantifying framework. More issues in tech are people problems, than technical. – nbering Jul 29 '18 at 2:15
  • 1
    Also, even an expert is liable to be subject to various kinds of bias, e.g., a particularly novel vulnerability might seem more dangerous than an objective analysis would justify, while a boring one might seem less dangerous than it should. This sort of formulae, while not by any means eliminating bias, might be helpful in reducing it. (Not an answer, because I'm only speculating: if someone has evidence that this is/is not true, that would be much more useful.) – Harry Johnston Jul 29 '18 at 2:24
1

CVSS scores are meant as a guideline not as the gospel.

Proper risk assessment for your environment should always be determined by a review of what is considered critical infrastructure, what is more AT RISK of being exploited, etc.

If i have a device that cannot be updated due to required vendor software for business functionality, it has 10 critical vulnerabilities with a rating of 10 each.

But that device is only plugged in once a month for 30 minutes at non-regular intervals. I would say that device is a low to medium risk for my network.

Now if I have a device that is in the DMZ and has multiple vulnerabilities that are easily exploitable but aren't considered a "Critical" threat rating. I would evaluate that as being higher risk because it is exposed to the public

Edit:

Good additional resources to check out from Rapid7 (creators of metasploit): changing criticality of an asset, Adjusting risk with criticality, Risk scoring FAQ

  • I think all these statements are true but not sure how it's an answer to my question. My question is: is there ever any advantage to using a formula to evaluating a security vulnerability (like the DREAD formula), compared to just having a security professional take everything into account and say, "I give this an 8 out of 10." – Bennett Oct 26 '18 at 18:46
  • The answer is you should use a formula and evaluate based on your environment – Matt Oct 26 '18 at 18:50
  • But that's the question: Why use a formula? Is there any evidence that this works better than just trying the security professional to assign a rating, without using a "formula"? – Bennett Oct 28 '18 at 3:25
0

I think you missed the point of the formula. The usefulness is not in the final score (that's for comparison to other vulnerabilities) but in the individual scores in each area.

As a professional, if you come to me with a vulnerability and tell me "yeah, it's a score of X", that's utterly useless to me. What I want to know are the component scores. Exploitable, discoverable, with high damage? Yeah, I want to get on that today. High damage that's difficult to discover and difficult to exploit? I'll patch in the next cycle.

DREAD and CVSS gives me those. How they come up with their final score is not interesting to me, but I want to know what the experts think in each area of the score. And I'm not going to assume that I am expert enough or that I have someone working for me that is expert enough to make that call. Tell me, high, medium, low, what the vuln is in each area.

  • That sounds right, but still, same question: Is there evidence that the formula works better than just having a security expert eyeball it and make a judgment call? Of course each individual component of DREAD is important. But I would assume that when an expert looks at a vulnerability and makes an instant judgment call, all of those factors are baked in intuitively. – Bennett Nov 25 '18 at 20:39
  • What evidence would suffice? It's inherently subjective and qualitative. – schroeder Nov 25 '18 at 20:40
  • Evidence would be an experiment where: two groups of researchers consider the same vulnerability and assessing the priority of fixing it. One group applies the formula. The control group just eyeballs it. At the end, ask: In which group does there seem to be more in-group agreement? And do the groups agree with each other, on average? If the groups agree on average, that suggests the formula doesn't matter. If the formula group has less in-group agreement, then the formula isn't even helping converge on any answer, much less the correct one. – Bennett Nov 25 '18 at 20:46
  • That process will not result in a repeatable experiment. My answer is about the uselessness of an overall score. You appear to only be concerned about the aggregate score where intuition is baked in. I say that no matter how you arrive at that score, it is meaningless. – schroeder Nov 25 '18 at 20:51
  • I don't know how this is not a "repeatable experiment". You can always repeat the experiment with the same vulnerability, and new groups of people, the same way you'd repeat any other experiment involving people. (You could also repeat the experiment with a different vulnerability -- perhaps some vulnerabilities, by their nature, result in more agreement.) (I should have been more clear: when I say "group", I still mean that the individuals all make their assessments individually, but the grouping is for comparison. Not for making a "group decision".) – Bennett Nov 26 '18 at 21:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.