White-box vs. Black-box

Question

What are the relative advantages and disadvantages of each form of testing?
I.e. What is the difference between static code analysis and runtime/dynamic penetration testing?
What are the pros and cons of each? Are there situations where one is preferable over the other?

---

Partially from the original Area51 proposal

Answer 1

I believe this question is best addressed in chapter 4 of The Art of Software Security Assessment, a book by Mark Dowd, Justin Schuh, and John McDonald.

Without it as a reference, I can safely tell you that the best method is using runtime data along with source-code while determining "hits" (or traces, aka coverage) using black-box testing -- but only after a threat-model and general architecture of the system is well-known.

The authors also appear to like secure static code analysis when combined with candidate-point strategies, although it's my opinion that these can vary wildly in value assuming the following are not true:

• The language, and its base class libraries, must be supported by the secure static analysis tool
• The entire system must usually be available. I.e. it must include buildable source code including all third-party/contrib components and external libraries -- possibly even to include the system compiler, VM, or other artifacts of the original system
• All external components/libraries that are not a part of the base class libraries must have sources and sinks defined in the secure static analysis tool's source-sink-passthru database. The intricacies of some passthrus (i.e. filters) can vary by implementation or implementer, and thus almost always require custom configuration
• Additional use of certain patterns or architectural code elements could cause other variations, which could require customization that isn't possible with most modern secure static analysis tools

For the reasons above, as well as the reasons put forth in the NIST SATE studies (done by NIST SAMATE), I find it difficult to recommend many secure static analysis tools for use in white box analysis. It is almost always simply better to use code comprehension strategies that most likely involve reading the source code from top to bottom, which is really very important if you are looking for managed code rootkits and the like.

Instead of testing and auditing/assessing applications, I would take a different approach that is largely very technology-agnostic. My suggestion would be to implement an application security risk management portal that includes an app inventory along with each app's currently implemented application security controls. After an initial baseline, the application security controls should be evaluated against industry standards such as MITRE CWE, SAFEcode, and OWASP ASVS. A gap analysis (note that this is a standard risk management term and works best when implemented in an information security management program based on ISO 27001 or similar) can then be used to determine the optimal application security controls, as well as a path to get from the currently implemented controls to the required ones.

You should implement this risk management portal before performing risk assessment activity such as white-box or black-box testing in order to get better results, and to measure the success of your program.

Answer 2

Black box

• pros: easy, quick and simple testing
• cons: sometimes it is not possible to test some parts of application (to check hashing algorithms, session id entropy,...); you cannot be sure if whole application was tested

White box

• pros: ability to check source codes (saves time - no need to test SQL injection if you can see that parameters are used in safe way everywhere); you can test parts of application which are not accessible/testable using GUI
• cons: tests can become really complex

In general white box testing allows you to dive into the source code and perform complete penetration testing, but can be very time consuming, while black box is easy, fast and simple. I prefer gray box testing - using black box methods and interviewing developers/checking source code only on specific parts of application (authentication, session management, configuration management, ...).

Answer 3

Some really good answers here, but a few additional points I think are important to add:

• As @atdre mentioned, it shouldn't be either / or, these are two different creatures, and they measure different things. If at all possible, you should do both.
• Also as @atdre said, testing - even whitebox + blackbox - is not enough. There are other things you need to do to be secure, including all of a holistic SDL, with proper risk management, analysis, etc.
• To the point... Blackbox is usually quicker, often by order of magnitude. Whitebox (including code review) usually requires a lot more work.
• Blackbox (i.e. pentesting) usually has a cheaper cost than whitebox, not just in total but also per-hour.
• There are more quality 3rd party blackbox providers than whitebox - not in total, but counting only those that really know what they're doing. (Or is that just my perception?)
• WB often finds much deeper vulnerabilities than BB.
• WB can often find faulty filters, that BB was not able to bypass (until knowing how the filter is constructed - another point towards gray box testing).
• There are many types of flaws that BB is not able to even test - e.g. audit logs, crypto flaws, backend hardening, etc.
• BB can test the entire system, with all the non-code protections in place (e.g. WAF, IPS, OS hardening, etc.) where WB is only on the application level (e.g. code/design etc). Note that this can go both ways - sometimes you get prevented from completing a BB scan, even though once you know the vector you would be able to bypass the protections.
• Similarly, BB can discover faulty interactions between sub-systems, where WB will usually miss it. Consider this as unit-testing vs. system-testing.
• WB can often be performed long before the system is complete, BB needs to be built, compiled, up and running (and preferably most functional bugs shaken out). This can make an SDL more efficient, when reviews can be done early in the lifecycle.
• On the other hand, if a system is already up, it's simple to start a BB, but if you want to do WB (do it right) you have to start hunting down all the source code, libraries, tools, etc. Often, you don't even have the source code because its 3rd party, COTS, etc.