How to preform black-box (no source) vulnerability testing/fuzzing?

Question

How would I (or) what software would I use to conduct local application (stored on the PC) black-box (no source/closed source) fuzzing or vulnerability testing for Windows? I've just been using common sense, like using negative numbers and such, directory traversal strings, all that. I'm comfortable with using Linux if necessary.

Also, what exactly is fuzzing? From my understanding, its inputting massive amounts of random data to a program, but I cannot find any usable "fuzzers" for local applications.

Answer 1

Fuzzing refers to random input testing. Contrast this to a scenario where you subject a program to a set of known inputs, which you might otherwise call unit testing. The basic idea is that you programmatically generate a large number of inputs to a program in hopes of finding some combination of inputs that causes incorrect behavior- either to detect bugs or to look for the basis of constructing an attack. There aren't really any general purpose fuzzing tools because fuzzing tends to be application specific and the inputs can't be completely random.

For example, suppose you've got a program that reads an XML file, performs some computation on that file, and then produces another file as output. To fuzz that program you could randomly generate XML files and submit it to the program. You can't necessarily know what output your program should produce (or else you wouldn't have needed the program), but you might say:

1. The program should always terminate within 10 seconds
2. The program should not crash
3. The program should always produce an output file, even if we can't verify the output file is correct
4. You might be able to verify some attributes of the output- for example if the XML input file has 27 top level tags then your output file should have 27 lines.

So you randomly generate a million XML files run your program a million times, and every time one of those conditions is not met you record what inputs caused the apparent problem. Then a developer can use those inputs to hopefully find whatever condition caused them.

Note that we're using random XML files, not totally random files. A totally random input file would be a random sequence of characters. There are a huge number of totally random files that do not constitute valid XML and won't test your program in any productive way. A totally random fuzzer would likely never generate a valid XML file to actually test the program logic. This is the basic reason why there is no general fuzzing program- random inputs need to be meaningful rather than totally and truly random in most cases.

Generating totally random invalid inputs can be meaningful, especially if you want to ensure that your program doesn't crash on mal-formed inputs. But this is essentially testing input validation and not program logic. You can use fuzzing to test both things.