How can reverse engineering (RE) help a security expert?
I've participated in CTF challenges and I have seen RE as an important part of CTFs. Can anyone explain or show me a real scenario that shows how RE or cracking can help hackers break security?


What do you do in most reverse engineering challenges in CTF's

(if you're not just pulling out passwords with strings)

  • You analyze the logic of the program in it's normal behavior.
  • Analyze which computations are done on our input
  • Find out where 'bad boy' and 'good boy' are located
  • Find a way to make our input lead to our 'good boy'

What do you do when looking for security bugs?

(in a fuzzing approach)

  • You analyze the logic of the program in it's normal behavior
  • Analyze which computations are done on our input
  • You analyze the logic of the program when given unexpected input
  • Analyze the behavior and see if we can exploit the new behavior

This is just one example

Actually, just reversing the program to see which function calls it makes can be enough to find a security bug, if you see a call to the unsafe strcpy for example, you know it's worth having a look at from a security perspective.

On top of that, reversing is done to understand program logic, if you want to break a program's logic and take advantage of that ( or protect the program ), you have to know exactly how it works.

Edit after your comment: As for how a hacker can actually use this to breach a security system; The strcpy example above implies a buffer overflow is present, this is basically the Hello World of exploit writing.

If you want an advanced guide on what that is and how it works, you could read these articles on the corelan blog which describe how buffer overflows work and how they can be exploited.

While reading those, you'll realize an exploit writer's language is really the same as a reverse engineer's language. Which is why it is helpful to train your RE skills to become a better Exploit writer.

Malware analysis

I almost dared to leave this answer here without noting Malware Analysis. Malware writers are becoming quite spectacular when it comes to hiding malicious activities from AV.

It's very important for AV Vendors to have people who can reverse engineer the extreme puzzles they sometimes set up in order to protect machines worldwide from sophisticated malware attacks.

  • Thanks @Nomad for your awesome answer! May I ask how can it help a hacker, break a system's security? – pyDeb Apr 14 '18 at 4:42
  • I have added some more info to the This is just one example section above to answer your comment! – Nomad Apr 14 '18 at 12:27
  • 1
    I'm really thankful for your complete answer ;-) and I'm sorry for answering this late, been busy with some projects :). Thanks again! @Nomad – pyDeb May 2 '18 at 6:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.