How do you start analysis? Do you start at main and spread out from there, or do you have a better method?
Start on exhausting basic analysis (both dynamic and static) - enumerate exports, imports, function use, syscalls, winapi, mutex, dll dependencies, strings and some grepping on that.
Run dynamic analysis on basic sandboxes to come to some, eventhough partial and can be somewhat wrong, you can come up by now with several theories on some of the main functions of executable/dll.
that said, if we are talking java/.net etc. - of course -decompile them, but there is no common practice on using malware on such environments.
If you spot function calls to something suspicious - let's say that the exec tries to write to some critical system files/registry values, or deploying strange named files - you should be worried (or happy, depending on your hat-color :) )
How do you find and identify important functionality, or particular functionality that you're interested in?
Strings can be helpful - you can spot something suspicious as a string starting with cmd.exe... or even hostnames, user-password combinations, and oters
Resource Hacker and dependancy walker are basic tools to enumerate exports, imports and resources included.
The most important functionality nearly always have to be reverse engineered in IDA or similar static analysis tool.
How do you map out high level control flow?
If all of the above fails, IDA's graphing capabilities are great and can be used for it.
How do you manage the helper routines you've identified? I find bookmarks to be insufficient, and notepad to be too primitive.
IDA have commenting system, coloring options, renaming and more.
For the overwhole process I like to graph things out when needed, it is the most clear way to do that - even on visio.
How do you avoid getting lost in the avalanche of assembly code?
You almost never rev-engineering on asm level ALL of the code available. Some are more efficient to do on dynamic analysis (Olly and Immunity are great - Immunity is an Olly fork with a lot of twists), and you never need all code to be reversed in order to figure it out.
I have a color coding in IDA and constantly renaming the already reversed parts to something more sensible than 'loc_402BBD'
Any other tricks / tips for approaching these kinds of tasks?
- Never get stuck on one mind-state, it can bring you a lot of trouble - think of analysis code for days and than getting some part that totally change the way you looked on things, horrible.
- Practice, a lot, there is nothing like it, believe me.