Is there any documented malicious API calls somewhere.. I couldn't find one in paper.
No, because all API calls can be both valid and malicious, depending on context. In terms of analysing a piece of malware, this is something you normally do by hand.
A quick primer on Windows executables looks like this: All windows executables that make API calls must import those addresses from System DLLs. This happens at two levels; statically, where the imports are coded in and dynamically, if the executable loads DLLs on the fly. So one place to start analysis is to look at what an executable imports in terms of API calls. You can do this with:
dumpbin /IMPORTS executablename.(exe|dll|ocx|scr|anything executable)
(alternatively, disassemblers will present this information in the output disassembly).
Another useful technique might be to dump all printable ascii strings and take a look through them. If the malware is poorly written, you might well find hard coded urls to suspicious locations in there or other data. If you find the string
cmd.exe in there, that's pretty suspicious.
Other suspicious properties of executables include abuses of the PE format, for example incorrect use of headers. Most if not all linkers will respect all the headers of the format, yet some are unnecessary and Windows will actually execute an image with code in the headers with careful assembly. Small executables like this, especially those that make calls to network functionality, should be viewed as suspicious.
Images with data beyond the end of the image (this is allowed - this is how winzip self extractors and installers of the exe variety typically work) could also be viewed as suspicious, since you have a data payload there.
However, a good malware author knows this and is more than capable of obfuscating those strings and providing valid executables as the actual size cost isn't that prohibitive. At this point, statically, it is often worth disassembling the code and reading through it. A thorough knowledge of assembly is necessary at this point, as is knowledge of how linking works and how to interact at an ABI level with the operating system you're on. Rakkhi mentions labs full of people - this is where they come in.
So that's a brief set of things you could do to analyse an image statically (without running it). If you have a system to sacrifice (like a virtual machine) then you might also consider running it and analysing what happens. Some tools for Windows:
depends.exe from the DDK will profile executables to work out all DLLs they import.
windbg.exe allows you to debug a live image running, telling you at run time what the various values in registers might be, what calls are being made. Some people like ollydbg, but it has no x64 support at the time of writing, so I don't use it.
- Various tools from SysInternals come in useful for finding out what registry keys, files etc an application is attempting to interact with. See Mark Russinovich's writeup of detecting the sony rootkit with exactly this set of tools.
- IDA Pro. I can't afford said tool myself, but it is one of the tools in this industry.
These are equivalent to commands like
ltrace on Linux, along with
pmap. They give you information about what the process actually tried to do and the debugging tools in there allow you to step it through. I should point out this is very unsafe and you should only really do it in a controlled environment.
In terms of actually running processes, there are a few things which generally look suspicious all the time. These don't necessarily mean the process itself is malicious - it could well have been exploited and had code injected. They are:
- mmap(0). No process should really need to mmap to zero; operating systems will usually map to any other address space under standard usage. mmaping to zero is useful for an assembler-based program because 0 is an easy reference address. Jonesforth does this, for example, so that any 0 bytefields under a debugger indicate code being interpreted from forth syntax rather than a built in function, which is a handy trick. By and large though, unless the executable was written entirely in assembly it will never do this. Malware authors might use it as an easy reference point, because there's probably nothing mapped there.
- executable stacks. Pages on the stack do not usually to be executable since the stack should only really contain data.
- executing data pages elsewhere. Very few programs need to load in custom code and execute it.
- Access to certain operating system resources, like
\Device\PhysicalMemory or the Linux equivalent
- For drivers, modifying the system service table (Linux calls this the sys call table). On Linux, the sys call table is marked as read only by default since 2.6.24ish. On Windows, Microsoft have promised they will continually attempt to prevent anyone modifying the system service table directly, since the driver based API provides all the functionality any legitimate code probably ever needs.
- Indiscriminate DLL injection, i.e. system wide dll injection. See also installing shell extensions, adding code to start up paths, boot execute modification - anything autoruns exposes could be both legitimate operation or malware attempting to persist reboot. If you're looking at a process either statically or dynamically, if it puts stuff here, do investigate what it does.
As for how AV does signature recognition, Rakkhi has that covered, particularly his observation that antivirus implementations act like malware in how they work (but not what they do). A signature only needs to consist of enough information to (hopefully, although false positives happen) uniquely identify the malware, so key properties of the file (size, say) and key characteristics in the file (certain combinations of instructions followed by zipped data for example).
From a career perspective I can't advise on how to get into this field; I'm an amateur. However, technically, if you're looking to work out how to examine a binary, I'd say pick on a safe one. Choose a program you have on your system, say
notepad.exe, and look at what it does and how it behaves. Then move on to something more complicated. I say this because you can set the system up once and know it won't be damaged this way, so you can improve your work without having to reconstruct the system because the malware made it impossible to use after your dynamic analysis went wrong and you let it clobber the system.
Hopefully that gives you an idea of how to get started. Really, this is a massive field and I'd need a couple of books to explain it thoroughly. Finally, be careful handling malware.
Some really great resources not already mentioned:
- x86 assembly wikibook. Surprisingly good resource on what you're reading.
- Agner Fog's assembly resources. Outstanding detail; I use objconv from here to disassemble programs as it is one of the best free disassemblers I've ever seen (even includes support for x64, SSE4, etc).
- Windows PE format, just for interest.
- Windows SDK, Windows DDK.
- Mark R (again) on Stuxnet, part 1. Uses SysInternals to analyse the Stuxnet virus.
- The Old New Thing - this is a blog on programming at the Win32 level from the guys at MSDN. It runs through how various parts of Windows programming work; following it over time you'll learn an awful lot.
- Really small ELF files for Linux. Just because it's cool. Goes nicely with the PE one.