Benefits of secure code review in-IDE vs. fatapp vs. webapp

Question

For those of you who have worked with commercial secure code review tools such as:

• Klocwork
• Coverity
• Armorize
• Fortify
• Checkmarx
• Appscan Source Edition (formerly Ounce)

Or perhaps a free or open-source equivalent such as:

• CAT.NET
• FindBugs
• Klocwork Solo
• Armorize demo
• Fotify Teamserver demo (or the Audit Workbench that comes with the book)
• OWASP O2 (Ounce Open)

do you generally prefer the in-DE plugins/addons (e.g. Fortify Secure Coding Plugin for Eclipse), the standalone fat app (e.g. Audit Workbench), a web application (e.g. Armorize, Fortify Teamserver), a complete build server solution, et al? Why?

Answer 1

Not much of an answer, but I would say it depends more on the feature-set of each product, and the usablity of each interface.
E.g. Fortify's IDE plugin, fatapp, and webapp are not equivalent in supported features. Even more, Fortify's Eclipse plugin is not equivalent to their own VisualStudio plugin!

In principle though, assuming all else being equal, my preference would be:

• As an auditor / security consultant, I would prefer the standalone. Dont need to start messing with their environment, helps me focus more, etc.
• As a developer (and for my clients/developers), I would prefer IDE plugin, so as to put everything right in front of them, thy dont need to open another tool, they wont forget about it, makes it part of their dev/compile, etc.
• A web app for statistics (ala what Fortify tried to do with their 360 server, but not great IMO) is great for management and such.

A note, the above mostly depends on relying on automated code scanners, as opposed to manual code review. Sometimes that's what is needed, but in general usually not good enough.

Answer 2

Both are necessary to choose one over the other depends on the development lifecycle approach: For instance, typically in a Scrum project, the IDE based ones makes more sense as you can add a backlog item asking for scanning code once per sprint or even once per day and that would be highly useful.

On the other hand if you are developing in Waterfall or Spiral, then going for standalone versions of the tools would be smarter choice to integrate many packages.

As AviD mentioned for other than developers the standalone version would be better

Answer 3

In having worked with Fortify as standalone, with its Maven plugin and a brief touch on its Team Foundation plugin I have a similar set of preferences to AviD:

• The standalone is lovely when you can be provided with the full codebase, it runs quickly and as long as you have all the dpendencies it just works!

• To implement a recurring test scenario the plugins are the way to go - have them built in so a compile and build automagically reviews for security issues.

• Management do need reporting and trending, so a management server is required.

tl;dr - what AviD said, pretty much:-)

Answer 4

Based on my experience, the standalone app is probably going to be the nicest for code auditors and code reviewers.

The advantage of the web-based interface comes for collaboration: if you have a team of folks who are all auditing an application, triaging and analyzing the static analysis warnings, entering in their analysis of whether the warning is a bug or not and its severity, etc. Web-based interfaces can be great for that.

I don't personally know what developers will prefer. I suspect an IDE plugin may have advantages, because of its ability to provide rapid feedback and its integration with the workflow the developer already uses, but I'm guessing here.