Many malware executables use hooking to be invisible. I am familiar with some hooking methods, I'm not sure what methods exist for detecting them. Is there a way to tell when a function call has been modified to perform other tasks through hooks?
Hooks are implemented in a whole bunch of ways:
- Modifying legitimate jump instructions to point at hooks instead of the normal code.
- User call table (IAT) hooking - modifying the addresses of user-mode APIs in a process.
- Kernel call table hooking (e.g. SSDT / GDT ) - replacing a call table pointer with the address of your hook.
- WndProc hooks (e.g.
PeekMessage) - hooking onto window notification messages.
- Legitimate callbacks like
I'm guessing you're most interested in the first two types.
Jump hooks can be created in a near-infinite number of ways. This makes it almost impossible to write a tool to identify the hooks. However, you can use integrity checking tricks, e.g. comparing code in the binary file (e.g.
dll) to the code in memory. You could also hook
WriteProcessMemory and other such APIs to detect modification of process memory, though this only works against user-mode attacks.
IAT hooks are a little easier to check for. Take a snapshot of the IAT of a process when it starts (e.g. from the static binary) and compare the in-memory IAT to the real addresses of the functions that should be in there. For example, if you know IAT entry 4 points to
user32.MessageBoxA, you can use
GetProcAddress to find the real address of that function and compare the address in the IAT to that. If they don't match, you know it's been hooked.