Suppose I have a binary which is suspicious. How can I determine whether it is malicious, and if it is, reverse-engineer how it works and what it does? For example, maybe using static analysis?
Are there any documented malicious API calls somewhere? I couldn't find any in the papers I've read. I could submit it to a sandbox, but I want to learn how to analyze a malicious PE binary.
Is there any documented malicious API calls somewhere.. I couldn't find one in paper.
No, because all API calls can be both valid and malicious, depending on context. In terms of analysing a piece of malware, this is something you normally do by hand.
A quick primer on Windows executables looks like this: All windows executables that make API calls must import those addresses from System DLLs. This happens at two levels; statically, where the imports are coded in and dynamically, if the executable loads DLLs on the fly. So one place to start analysis is to look at what an executable imports in terms of API calls. You can do this with:
dumpbin /IMPORTS executablename.(exe|dll|ocx|scr|anything executable)
(alternatively, disassemblers will present this information in the output disassembly).
Another useful technique might be to dump all printable ascii strings and take a look through them. If the malware is poorly written, you might well find hard coded urls to suspicious locations in there or other data. If you find the string cmd.exe in there, that's pretty suspicious.
Other suspicious properties of executables include abuses of the PE format, for example incorrect use of headers. Most if not all linkers will respect all the headers of the format, yet some are unnecessary and Windows will actually execute an image with code in the headers with careful assembly. Small executables like this, especially those that make calls to network functionality, should be viewed as suspicious.
Images with data beyond the end of the image (this is allowed - this is how winzip self extractors and installers of the exe variety typically work) could also be viewed as suspicious, since you have a data payload there.
However, a good malware author knows this and is more than capable of obfuscating those strings and providing valid executables as the actual size cost isn't that prohibitive. At this point, statically, it is often worth disassembling the code and reading through it. A thorough knowledge of assembly is necessary at this point, as is knowledge of how linking works and how to interact at an ABI level with the operating system you're on. Rakkhi mentions labs full of people - this is where they come in.
So that's a brief set of things you could do to analyse an image statically (without running it). If you have a system to sacrifice (like a virtual machine) then you might also consider running it and analysing what happens. Some tools for Windows:
depends.exe from the DDK will profile executables to work out all DLLs they import.windbg.exe allows you to debug a live image running, telling you at run time what the various values in registers might be, what calls are being made. Some people like ollydbg, but it has no x64 support at the time of writing, so I don't use it.These are equivalent to commands like strace and ltrace on Linux, along with pmap. They give you information about what the process actually tried to do and the debugging tools in there allow you to step it through. I should point out this is very unsafe and you should only really do it in a controlled environment.
In terms of actually running processes, there are a few things which generally look suspicious all the time. These don't necessarily mean the process itself is malicious - it could well have been exploited and had code injected. They are:
\Device\PhysicalMemory or the Linux equivalent /dev/kmem.As for how AV does signature recognition, Rakkhi has that covered, particularly his observation that antivirus implementations act like malware in how they work (but not what they do). A signature only needs to consist of enough information to (hopefully, although false positives happen) uniquely identify the malware, so key properties of the file (size, say) and key characteristics in the file (certain combinations of instructions followed by zipped data for example).
From a career perspective I can't advise on how to get into this field; I'm an amateur. However, technically, if you're looking to work out how to examine a binary, I'd say pick on a safe one. Choose a program you have on your system, say notepad.exe, and look at what it does and how it behaves. Then move on to something more complicated. I say this because you can set the system up once and know it won't be damaged this way, so you can improve your work without having to reconstruct the system because the malware made it impossible to use after your dynamic analysis went wrong and you let it clobber the system.
Hopefully that gives you an idea of how to get started. Really, this is a massive field and I'd need a couple of books to explain it thoroughly. Finally, be careful handling malware.
Some really great resources not already mentioned:
To statically analyse a program on the assembly level to figure out if it is malicious sounds very very difficult. If you could do that, I imagine you would be able to find any bug in an executable also. If you would just search for dangerous calls, like file system accesses, you'd find many many programs doing that without being malicious. Using the file system is a common thing for most programs. So, one would need to have another level analyzing what the program actually is doing, and identify any malicious actions. But this analysis needs to work in any depth of complexity, as source code can easily be intertwined, etc., and the compilers can shift code around, making it hard for an analysis to find the chain of events for a malicious attack. And the next step, what malicious attacks should be searched for? And how can one define each and every attack in a way that it will be found with such an elaborate analysis method?
If I understand correctly, all the current anti-virus methods are based on recognizing patterns from identified viruses, which is a much easier way to find malicious programs.
This question makes me think of the writing of Douglas R. Hofstadter in Gödel-Escher-Bach, and the self-replicating part. I think it was something in the line of - "Any powerful enough system will be capable in destroying itself".
In short, if we could find a (new) malicious program just by looking at it on the disassembly level, I imagine the OS and development tools would be too simplified that the system is hardly usable. If the OS and development tools were advanced enough, the system and tools would be able to be malicious in ways that with our limited knowledge we wouldn't find it (early).
There are entire labs of people at antivirus vendors doing this. Easiest would be to submit it to one. Sophos, McAfee, Symantec, etc.
From a number of presentations I have seen, analysis typically involves looking for unusual behaviour or content in the binary. Behaviour ironically looks like AV behaviour. Hooks into system calls, hijacking browser objects, dialing home etc. Hiding content can be injecting executables into an Excel file or movie for example. Decompiling the binary, assure with identifying these.
AV works in a similar manner, monitoring processes and memory for unusual activity and examining files for known signatures (usually this would be an individual file level rather than every line in every binary).
Specific malware forums will give you more information and resources for study if you are looking to get into the field.
The simple and easy solution is: submit your sample to Virus Total and see what they say. They'll run dozens of anti-virus products on it and share with you the results.
For web-based malware (including PDF files), I recommend Wepawet.
Funny I was just attending a talk given by Aubrey-Derrick Schmidt from the DAI Labor in Berlin, who presented such a set of method calls for Symbian OS. I am not 100% sure it helps but give it a try and check out Static smartphone maleware detection.
Basically Aubrey used IDA Pro on a large set of different Malware (Downloaded through piratebay) and then applied a statistical comparison.
