Suppose I have a binary which is suspicious. How can I determine whether it is malicious, and if it is, reverse-engineer how it works and what it does? For example, maybe using static analysis?

Are there any documented malicious API calls somewhere? I couldn't find any in the papers I've read. I could submit it to a sandbox, but I want to learn how to analyze a malicious PE binary.

  • 2
    please try to stick to one question per question. If you ask multiple questions, it decreases the likelihood of getting a good answer. If you have multiple questions, I suggest posting them separately.
    – D.W.
    Jun 26, 2011 at 5:10
  • can you clarify what you mean by "detect its maliciousness"? Do you mean "determine whether it is malicious"? Do you mean "reverse-engineer it to determine how it works"? something else? What's the purpose, and what problem are you trying to solve?
    – D.W.
    Jun 26, 2011 at 5:10
  • Hi @talfiq, welcome to the site! Thanks for the interesting question... But I'd like to point out that "malware" and virus do not necessarily work the same - and, antimalware / antivirus work in very different ways.
    – AviD
    Jun 26, 2011 at 9:13
  • 1
    While @ninefingers has provided a great answer, I've stuck a bounty on in hopes of getting some reaction from real-life malware analysts too.
    – user185
    Jun 27, 2011 at 21:11
  • 1
    @talfiq yes, that is correct of course, malware is the generic term for, well, malicious software :). However, thanks to the vendors - anti-malware is not a superset of anti-virus, it works differently - the term is also often used exclusively of standard virii, worms, etc.
    – AviD
    Aug 26, 2011 at 10:45

5 Answers 5


Is there any documented malicious API calls somewhere.. I couldn't find one in paper.

No, because all API calls can be both valid and malicious, depending on context. In terms of analysing a piece of malware, this is something you normally do by hand.

A quick primer on Windows executables looks like this: All windows executables that make API calls must import those addresses from System DLLs. This happens at two levels; statically, where the imports are coded in and dynamically, if the executable loads DLLs on the fly. So one place to start analysis is to look at what an executable imports in terms of API calls. You can do this with:

dumpbin /IMPORTS executablename.(exe|dll|ocx|scr|anything executable)

(alternatively, disassemblers will present this information in the output disassembly).

Another useful technique might be to dump all printable ascii strings and take a look through them. If the malware is poorly written, you might well find hard coded urls to suspicious locations in there or other data. If you find the string cmd.exe in there, that's pretty suspicious.

Other suspicious properties of executables include abuses of the PE format, for example incorrect use of headers. Most if not all linkers will respect all the headers of the format, yet some are unnecessary and Windows will actually execute an image with code in the headers with careful assembly. Small executables like this, especially those that make calls to network functionality, should be viewed as suspicious.

Images with data beyond the end of the image (this is allowed - this is how winzip self extractors and installers of the exe variety typically work) could also be viewed as suspicious, since you have a data payload there.

However, a good malware author knows this and is more than capable of obfuscating those strings and providing valid executables as the actual size cost isn't that prohibitive. At this point, statically, it is often worth disassembling the code and reading through it. A thorough knowledge of assembly is necessary at this point, as is knowledge of how linking works and how to interact at an ABI level with the operating system you're on. Rakkhi mentions labs full of people - this is where they come in.

So that's a brief set of things you could do to analyse an image statically (without running it). If you have a system to sacrifice (like a virtual machine) then you might also consider running it and analysing what happens. Some tools for Windows:

  • depends.exe from the DDK will profile executables to work out all DLLs they import.
  • windbg.exe allows you to debug a live image running, telling you at run time what the various values in registers might be, what calls are being made. Some people like ollydbg, but it has no x64 support at the time of writing, so I don't use it.
  • Various tools from SysInternals come in useful for finding out what registry keys, files etc an application is attempting to interact with. See Mark Russinovich's writeup of detecting the sony rootkit with exactly this set of tools.
  • IDA Pro. I can't afford said tool myself, but it is one of the tools in this industry.

These are equivalent to commands like strace and ltrace on Linux, along with pmap. They give you information about what the process actually tried to do and the debugging tools in there allow you to step it through. I should point out this is very unsafe and you should only really do it in a controlled environment.

In terms of actually running processes, there are a few things which generally look suspicious all the time. These don't necessarily mean the process itself is malicious - it could well have been exploited and had code injected. They are:

  • mmap(0). No process should really need to mmap to zero; operating systems will usually map to any other address space under standard usage. mmaping to zero is useful for an assembler-based program because 0 is an easy reference address. Jonesforth does this, for example, so that any 0 bytefields under a debugger indicate code being interpreted from forth syntax rather than a built in function, which is a handy trick. By and large though, unless the executable was written entirely in assembly it will never do this. Malware authors might use it as an easy reference point, because there's probably nothing mapped there.
  • executable stacks. Pages on the stack do not usually to be executable since the stack should only really contain data.
  • executing data pages elsewhere. Very few programs need to load in custom code and execute it.
  • Access to certain operating system resources, like \Device\PhysicalMemory or the Linux equivalent /dev/kmem.
  • For drivers, modifying the system service table (Linux calls this the sys call table). On Linux, the sys call table is marked as read only by default since 2.6.24ish. On Windows, Microsoft have promised they will continually attempt to prevent anyone modifying the system service table directly, since the driver based API provides all the functionality any legitimate code probably ever needs.
  • Indiscriminate DLL injection, i.e. system wide dll injection. See also installing shell extensions, adding code to start up paths, boot execute modification - anything autoruns exposes could be both legitimate operation or malware attempting to persist reboot. If you're looking at a process either statically or dynamically, if it puts stuff here, do investigate what it does.

As for how AV does signature recognition, Rakkhi has that covered, particularly his observation that antivirus implementations act like malware in how they work (but not what they do). A signature only needs to consist of enough information to (hopefully, although false positives happen) uniquely identify the malware, so key properties of the file (size, say) and key characteristics in the file (certain combinations of instructions followed by zipped data for example).

From a career perspective I can't advise on how to get into this field; I'm an amateur. However, technically, if you're looking to work out how to examine a binary, I'd say pick on a safe one. Choose a program you have on your system, say notepad.exe, and look at what it does and how it behaves. Then move on to something more complicated. I say this because you can set the system up once and know it won't be damaged this way, so you can improve your work without having to reconstruct the system because the malware made it impossible to use after your dynamic analysis went wrong and you let it clobber the system.

Hopefully that gives you an idea of how to get started. Really, this is a massive field and I'd need a couple of books to explain it thoroughly. Finally, be careful handling malware.

Some really great resources not already mentioned:

  • x86 assembly wikibook. Surprisingly good resource on what you're reading.
  • Agner Fog's assembly resources. Outstanding detail; I use objconv from here to disassemble programs as it is one of the best free disassemblers I've ever seen (even includes support for x64, SSE4, etc).
  • Windows PE format, just for interest.
  • Windows SDK, Windows DDK.
  • Mark R (again) on Stuxnet, part 1. Uses SysInternals to analyse the Stuxnet virus.
  • The Old New Thing - this is a blog on programming at the Win32 level from the guys at MSDN. It runs through how various parts of Windows programming work; following it over time you'll learn an awful lot.
  • Really small ELF files for Linux. Just because it's cool. Goes nicely with the PE one.
  • Thanks Ninefingers for taking your time to respond with very long answers. Very helpful indeed ;)
    – talfiq
    Jun 26, 2011 at 4:04
  • 1
    Very nice answer. Pitty you don't know that IDA Pro has a 5.0 freeware version it includes support for x86 and PE. That should pad your toolkit.
    – this.josh
    Jun 27, 2011 at 5:53
  • One should note that executing data pages is a normal feature of JIT compilers, so this is becoming quite common in "honest" applications (e.g. a Web browser does that, for Javascript). Jul 7, 2011 at 16:29
  • @Thomas very true. In the interest of simplicity (so far as it is possible with this sort of answer), I avoided that area entirely, but yes, it can be legitimate.
    – user2213
    Jul 7, 2011 at 16:38

To statically analyse a program on the assembly level to figure out if it is malicious sounds very very difficult. If you could do that, I imagine you would be able to find any bug in an executable also. If you would just search for dangerous calls, like file system accesses, you'd find many many programs doing that without being malicious. Using the file system is a common thing for most programs. So, one would need to have another level analyzing what the program actually is doing, and identify any malicious actions. But this analysis needs to work in any depth of complexity, as source code can easily be intertwined, etc., and the compilers can shift code around, making it hard for an analysis to find the chain of events for a malicious attack. And the next step, what malicious attacks should be searched for? And how can one define each and every attack in a way that it will be found with such an elaborate analysis method?

If I understand correctly, all the current anti-virus methods are based on recognizing patterns from identified viruses, which is a much easier way to find malicious programs.

This question makes me think of the writing of Douglas R. Hofstadter in Gödel-Escher-Bach, and the self-replicating part. I think it was something in the line of - "Any powerful enough system will be capable in destroying itself".

In short, if we could find a (new) malicious program just by looking at it on the disassembly level, I imagine the OS and development tools would be too simplified that the system is hardly usable. If the OS and development tools were advanced enough, the system and tools would be able to be malicious in ways that with our limited knowledge we wouldn't find it (early).

  • the possible way is by analyzing the dynamic traces - system calls :)
    – talfiq
    Jan 22, 2013 at 17:02

There are entire labs of people at antivirus vendors doing this. Easiest would be to submit it to one. Sophos, McAfee, Symantec, etc.

From a number of presentations I have seen, analysis typically involves looking for unusual behaviour or content in the binary. Behaviour ironically looks like AV behaviour. Hooks into system calls, hijacking browser objects, dialing home etc. Hiding content can be injecting executables into an Excel file or movie for example. Decompiling the binary, assure with identifying these.

AV works in a similar manner, monitoring processes and memory for unusual activity and examining files for known signatures (usually this would be an individual file level rather than every line in every binary).

Specific malware forums will give you more information and resources for study if you are looking to get into the field.

  • 1
    It looks like McAfee took down their free submission tool, so you need to use the support portal.
    – Scott Pack
    Jun 25, 2011 at 13:38

The simple and easy solution is: submit your sample to Virus Total and see what they say. They'll run dozens of anti-virus products on it and share with you the results.

For web-based malware (including PDF files), I recommend Wepawet.

  • he says an unknown virus, I guess he meant zero day. If it is zero day then its signiture is new. so virus total most probably wont help
    – smttsp
    Jan 17, 2013 at 20:05

Funny I was just attending a talk given by Aubrey-Derrick Schmidt from the DAI Labor in Berlin, who presented such a set of method calls for Symbian OS. I am not 100% sure it helps but give it a try and check out Static smartphone maleware detection.

Basically Aubrey used IDA Pro on a large set of different Malware (Downloaded through piratebay) and then applied a statistical comparison.

  • 2
    Can you share some of the techniques here, instead of just linking to another site?
    – AviD
    Jun 28, 2011 at 9:27
  • Well he told me, he used IDA Pro on a large set of Malware (different families) and then made an statistical comparison. Sorry do not know any more details, but if you are interested in the subject just contact him.
    – joecks
    Jul 4, 2011 at 14:17
  • 1
    Thanks @joecks, can you put that (and any other information you have) in the answer? See the FAQ, and How to Answer.
    – AviD
    Jul 4, 2011 at 18:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.