What methodologies are useful when reverse-engineering malware / shellcode?

Question

Whilst I'm familiar with reverse engineering techniques from a technical standpoint, I've never found a good methodology for approaching and dissecting malware / shellcode. I've found hundreds of tutorials on the technical aspects of analysing malware, but very little on the actual methodology of approach.

A few questions to help you understand what I'm looking for:

1. How do you start analysis? Do you start at main and spread out from there, or do you have a better method?
2. How do you find and identify important functionality, or particular functionality that you're interested in?
3. How do you map out high level control flow?
4. How do you manage the helper routines you've identified? I find bookmarks to be insufficient, and notepad to be too primitive.
5. How do you avoid getting lost in the avalanche of assembly code?
6. Any other tricks / tips for approaching these kinds of tasks?

Answer 1

How do you start analysis? Do you start at main and spread out from there, or do you have a better method?

Start on exhausting basic analysis (both dynamic and static) - enumerate exports, imports, function use, syscalls, winapi, mutex, dll dependencies, strings and some grepping on that. Run dynamic analysis on basic sandboxes to come to some, eventhough partial and can be somewhat wrong, you can come up by now with several theories on some of the main functions of executable/dll.

that said, if we are talking java/.net etc. - of course -decompile them, but there is no common practice on using malware on such environments.

If you spot function calls to something suspicious - let's say that the exec tries to write to some critical system files/registry values, or deploying strange named files - you should be worried (or happy, depending on your hat-color :) )

How do you find and identify important functionality, or particular functionality that you're interested in?

Strings can be helpful - you can spot something suspicious as a string starting with cmd.exe... or even hostnames, user-password combinations, and oters Resource Hacker and dependancy walker are basic tools to enumerate exports, imports and resources included. The most important functionality nearly always have to be reverse engineered in IDA or similar static analysis tool.

How do you map out high level control flow?

If all of the above fails, IDA's graphing capabilities are great and can be used for it.

How do you manage the helper routines you've identified? I find bookmarks to be insufficient, and notepad to be too primitive.

IDA have commenting system, coloring options, renaming and more. For the overwhole process I like to graph things out when needed, it is the most clear way to do that - even on visio.

How do you avoid getting lost in the avalanche of assembly code?

You almost never rev-engineering on asm level ALL of the code available. Some are more efficient to do on dynamic analysis (Olly and Immunity are great - Immunity is an Olly fork with a lot of twists), and you never need all code to be reversed in order to figure it out. I have a color coding in IDA and constantly renaming the already reversed parts to something more sensible than 'loc_402BBD'

Any other tricks / tips for approaching these kinds of tasks?

1. Never get stuck on one mind-state, it can bring you a lot of trouble - think of analysis code for days and than getting some part that totally change the way you looked on things, horrible.
2. Practice, a lot, there is nothing like it, believe me.

Answer 2

One method which I love is to use a API Monitoring software. My favorite is the one by Rohitab (google it) . I debug the malware in question and step through while I monitor API calls made on the software. This will provide you with a lot of information on what the malware is doing. For ex: monitoring FileRead and FileWrite calls will let you know what files the malware creates or reads. Note that this is somewhat of a gray box technique and not purely reverse engineering. Although, if you find something interesting in the API monitor, you will know in which part of the assembly the interesting part is.

Cheers.

Answer 3

Ad 1: It doesnt matter how do you start, it's pretty much thing of preference. Normally you start from main(), as there are no other options if you consider actually stepping thru the code. Also matters what do you mean by analysis, if this is stepping in this case, main() is a good choice.

Ad 2: See ad 4.

Ad 3: If it's written in assembly, most often there is main block with jumps to all functions one by one. If it's written in C, there is also main block which is used in similar way, as malware is very often small piece of software.

Ad 4: By labelling all addresses to function names

Ad 5: see ad 4

Ad 6: You can use the following things:

• Disassembler labelling all calls to kernel and OS libraries automatically, so it's clear what is going on on these level.
• Virtual machine to execute code - qemu, wine are helpful in this. Not to isolate, but actually to track the malware activity in a precise manner, when debugger cannot be used. KVM is also a good decent choice.
• Identify fragments of code and it's function, e.g. understand the compiler and malware author techniques, and with this knowledge, going thru this is way much easier.

Generally, the hardest part is to start, but once a major part is analysed, every second piece of code is easier. By understanding various compilers and languages and the way they produce the machine code, it's far more easier to do.

Also, having the knowledge database of code fragments and compiler machine code outputs helps a lot. This is long way task and doing this without having frameworks is very hard, especially because there are so many compilers today (new versions), it's harder then ever.