0

Using SAST / SCA tools within the delivery pipelines is quite common these days; however, in the software my teams are building, the SAST tools that we're using are very rarely finding even relatively important security gaps - for the most part it's false positives that merely slow us down in the integration/deployment process as depending on the score, we might want to analyze the issue before moving further.

Now, our dev teams are quite educated in terms of security, and the setup is relatively buttoned up, mostly based on Spring and React, with some other security checks in place like secure coding requirements, code reviews with security aspect in them, or 3rd party deps vuln checks, so it might be that SAST is more useful in more junior teams, or teams that write software in less secure setup--or maybe we're just using the wrong SAST tools.

I couldn't find a solid, non-SAST-vendor-run research of the usefulness of these tools, some comparison of them, and some attempt at calculating if the gains from using them do exceed costs of using them ($$$ spent on licenses, time spent on maintenance, time invested into reviewing false positives, cost of slower/more engaging integration/deployments, etc.).

Is there any proper research out there that I couldn't find? Or even solid anecdotal evidence that makes you feel using SAST / SCA pays off in your teams? Or did we all just drink the kool aid and keep blindly using SAST tools, even though they generate negative ROI?

Improve this question
3
  • 1
    I’m not sure if this is even answerable in such a broad sense. Different tools perform differently in terms of precision, recall, accuracy etc. Some tools can be fine-tuned for the specific use case, some cannot. The usefulness will therefore vary a lot. Before giving up on SAST altogether, I would try to find out why exactly the tool is reporting non-vulnerabilities (or missing actual vulnerabilities). Can it be fixed with a different configuration or ruleset? Then try that. Or is it a fundamental limitation of the analyses used by the tool? Then try alternatives which may be more suitable.
    – Ja1024
    Commented Sep 2 at 23:00
  • 1
    You say you have "code reviews with security aspect in them". Do you believe humans reading code is better and more accurate than using static analysis to help guide the code review and flag potentially insecure code? You say you have "3rd party deps vuln checks". How does this differ from software composition analysis? You say that you have "secure coding requirements". How do you verify the implementation of those requirements?
    – Thomas Owens
    Commented Sep 2 at 23:28
  • 1
    the proof is in the pudding... and the pudding is black-box testing. These tools could help inform the black-boxers of possible test cases. The engineers should spend their time fixing actual vulnerabilities/bugs confirmed via real black-box testing that simulates the end-user as closely as possible. (imho, etc...)
    – browsermator
    Commented Sep 4 at 18:05

1 Answer 1

Reset to default
1

I don't have any "solid" evidence on ROI and frankly I'd be pretty skeptical of any such evidence anyway. But my experiences are such that I am convinced of the value of these tools, when used well.

Let's start with SCA, because it's more straightforward. The nature of SCA means that false positives are rare. Even if you're not using a dependency in a way that exposes a vulnerability today, do you want to take the risk that that won't change in the future? In reality it's easier to address all serious vulnerabilities that are identified by your SCA tool (assuming fix versions are available). So the effort here is fairly low and the payoff is good in the sense that you avoid carrying known vulnerabilities in your app that can be exploited.

SAST can be a bit trickier. If your team are spending lots of time fixing or ignoring false positives then clearly that's not a good use of their time. A good tool should allow for configuration that reduces the rate of false positives while still catching serious issues. Integrating such a tool with your IDE, or at least as part of your pre-commit checks, should also reduce overall effort. In my experience this remaining effort is worth it because often delivery pressures lead to mistakes creeping in, even in experienced teams with a good level of security awareness.

So in summary, SCA is a no brainer and the key to success with SAST is to keep effort low by checking early and minimising false positives.

Improve this answer
3
  • If you're not using a dependency in a way that exposes a vulnerability today, doesn't that mean that it is a false positive? Your application is not vulnerable but the SCA alerts anyway.
    – Sjoerd
    Commented Sep 3 at 9:29
  • @Sjoerd yes that is one way of looking at it. My point here is that it's still a risk because in the future you might use that dependency in a way that does expose the vulnerability. In my view it's often easier to treat as a true positive in the first place and just upgrade.
    – andycaine
    Commented Sep 3 at 10:08
  • On the subject of SCA, the tools I've used flag vulnerabilities and the more general case of out-of-date vulnerabilities. Knowing when you can update your vulnerabilities and keep using a supported version that will get patches if a vulnerability is found is just as important as knowing when a dependency has a vulnerability or when a patch for a vulnerability is made available.
    – Thomas Owens
    Commented Sep 3 at 10:37

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .