Is running an automated application vulnerability scanner sufficient to certify our application is secure? Why isn't it?

From the original Area51 proposal.


4 Answers 4


Sure automated checks are not sufficient, and believed that never will be. Well, until some AI is made. As for example fuzzers - they can cover only checks that fails for harsh input. Logical bugs will be missed. Source code analysis shows good results when the talk is about really severe bugs and those vulnerabilities that requires several conditions to meet. Binary analysis might be daunting for not so experienced person, but it is possible to inspect closed source code application and besides, it takes into account compiler specifications that are impossible to diagnose during source code analysis. So, every method has it advantage and should not be used alone.

  • 5
    I agree with this answer. In addition - using automated tools, you won't be able to check the security flaws in application business logic. Combination of automated and manual testing is IMO the correct way. For example in OWASP Application Security Verification Standard, automated testing is just the first level of cheking if application is secure: code.google.com/p/owasp-asvs/wiki/Approach
    – bretik
    Nov 12, 2010 at 14:12

Relying only on automated vulnerability scanning to vet the security of most applications is woefully inadequate.

Why? Because automated tools will miss vulnerabilities (type II errors/false negatives).

The uglier implication of this question is the application’s security is important yet it was ignored during development.

  • 1
    But of course it was ignored, yet can you elaborate for the benefit of those that might have asked the question?
    – AviD
    Nov 14, 2010 at 9:11
  • If an automated tool misses a vulnerability, isn't that a Type I error: i.e. false negative?
    – atdre
    Nov 14, 2010 at 12:39
  • 1
    Or perhaps you mean a Type III error i.e. a "false" false positive?
    – atdre
    Nov 14, 2010 at 12:39
  • atdre, yes, i meant to say false negatives, i will fix Nov 14, 2010 at 17:05

Applications cannot be certified as secure in the same way that vehicles cannot be certified as safe to drive. Your mechanic may hand you a list of things that he or she has checked and perhaps even verified as safe. However, that does not mean that something will not go wrong once you get your car on the road or track.

Only mechanics that seek to rip you off will provide a fully automated safety inspection. It would be insane to think that in the next 50 years it would be ok to consider any application fully secure or safe, even with artificial intelligence. Even formal methods with formal specifications have been known to fail.

I suggest that you properly scope an application's unique situation before you determine the outcomes.

  • 1
    Which I think in effect answers the question perfectly in that what you are saying is that it is impossible to rely on only automated vulnerbility testing software. Nov 16, 2010 at 23:23
  • I, personally, feel that automated vulnerability scanners, whether the network, application, or code kind -- all miss like 99.9% of vulnerabilities and report information about false vulnerabilities (or false information on them) 99.9% of the time. The only data I have to back this up is the NIST SATE data from 2008 and 2009
    – atdre
    Nov 17, 2010 at 21:52

Sufficient for what?

Sufficient to defend against other purely automated attacks like worms? Probably.

Sufficient to defend against targeted attacks by skilled attackers? Certainly not.

A better question is "Does an automated scan mitigate risk to an acceptable level for my application?" The answer is, of course, it depends...

To use an analogy, you wouldn't ask the question, "Is going to my General Practitioner sufficient?" without the context of "I need a yearly checkup" or "I just chainsawed off half my torso."

An automated scan might be sufficient for a low criticality brochureware site on its own domain, but is definitely not sufficient for a banking app. Most sites fall somewhere in the middle, so... It depends ;-)


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .