I've been asked this question, and I really don't know how would I do it. My answer was to take known vulnerabilities and try to exploit them on a different way, but this is not really a new vulnerability, rather a new attack based on a known vulnerability.

So my question is, how is the process of researching vulnerabilities? (In a piece of software, or on a system, or in a protocol.) I guess it would be something like:

  1. In-depth research about where you are trying to find a vulnerability. For example: in-depth research on how a PKI works.
  2. Once you know everything, find if there is any weak point and understand why it is a weak point.
  3. Try to exploit these weak points.
  4. Report.

Is that process right?

  • 3
    This is a very broad philosophical question with tons of books written on the logic of discovery. Look for books by Popper, Lakatos, Kuhn. – Deer Hunter May 31 '13 at 9:14
  • 1
    @DeerHunter The question isn't about discovery in general, it's about discovery of security vulnerabilities. While broad, I think it can be answered with an introduction to the subject. I posted a feeble attempt, I'm sure others here who do this for a living can provide better insights. – Gilles 'SO- stop being evil' May 31 '13 at 10:06
  • @Gilles - okay by me, was simply making sure the bases are covered as well. Was specifically thinking about the dark science of crypto - since the OP did not make it clear what vulnerabilities were meant. – Deer Hunter May 31 '13 at 10:49
  • Was also confused by the lofty word "research" which in my book applies to looking for something nobody else has done yet. – Deer Hunter May 31 '13 at 11:00

Paraphrasing someone who does this for a living: “Exploiting vulnerabilities is a craft. Researching vulnerabilities is a black art.”

A common way of looking for vulnerabilities in software (or hardware or a combination) is fuzzing. Look at some interface and throw all kinds of garbage at it. If it keeps operating correctly (“access denied”, “syntax error”, …), try some different garbage. If it does something interesting (no response, “segmentation fault”, …), you've hit something that isn't working as intended, so investigate further to see if you've found a real problem. As soon as you find that the target isn't working as intended, and the difference in behavior has an impact on security, you've found a vulnerability, and the next step is exploitation (leveraging the vulnerability to obtain some useful result that violates the security policy).

While you can fuzz with random garbage, it's more productive to look for boundary conditions. Try sending some null bytes, or some malformed unicode, or a name containing a single quote, or a length that's just above or just under the size limit.

If you have the source code, you can perform some static analysis on it and look for common issues (unchecked array accesses, free-form strings concatenated to form a database query, …). You can also perform static analysis on a binary but it tends to be a lot harder.

To look for vulnerabilities in protocols, you can fuzz the protocol in a test implementation. Another approach is to attempt to prove some desired properties of the protocol. If you get stuck in the proof, maybe that's because that desired security property in fact does not hold.

Sometimes you can find vulnerabilities by accident, because you happen to hit a corner case and it doesn't behave as intended and you find that this has an impact on security. But it is a lot more productive to have a thorough understanding of what you're attacking. You need to know which components to look at, how to tune your fuzzer to get interesting results this century, that your proof isn't just failing because you can't find the right arguments…

| improve this answer | |
  • I always had seen fuzzing as a way of automate known attacks, rather than a way of discover new flaws. – Ay0 May 31 '13 at 12:38

I use a lot of the basic UNIX tools. grep, awk, cut, head, tail, cat and vim. This is when the source code is there. If not, I hit IDA =)

Basically what I do is either search for functions that get implemented wrong often (memcpy, malloc, open those kinds of functions). And scroll back till I get the origin of the variables used as arguments. When these are controlled (filtered) I check if the filters are decent. If they are uncontrolled (user input) I start the exploitation process. And try to solve the puzzle on how to get the most out of that single function.

The other method I try is just hack into it. Meaning, I see a Search Box I just try the most basic xss possible and hope it works. Or with this one case I did I actually had a client application. That didn't need authentication, so I reversed it and showed that is not the way to go. I noticed the vulnerability being a security guy, those guys were developers that didn't care enough about security to know that reversing software was possible.

Those are my ways, Fuzzing is great too. As is mona.py

| improve this answer | |

Not the answer you're looking for? Browse other questions tagged or ask your own question.