Many malware executables use hooking to be invisible. I am familiar with some hooking methods, I'm not sure what methods exist for detecting them. Is there a way to tell when a function call has been modified to perform other tasks through hooks?
Hooks are implemented in a whole bunch of ways:
I'm guessing you're most interested in the first two types.
Jump hooks can be created in a near-infinite number of ways. This makes it almost impossible to write a tool to identify the hooks. However, you can use integrity checking tricks, e.g. comparing code in the binary file (e.g.
IAT hooks are a little easier to check for. Take a snapshot of the IAT of a process when it starts (e.g. from the static binary) and compare the in-memory IAT to the real addresses of the functions that should be in there. For example, if you know IAT entry 4 points to