Many malware executables use hooking to be invisible. I am familiar with some hooking methods, I'm not sure what methods exist for detecting them. Is there a way to tell when a function call has been modified to perform other tasks through hooks?

  • Heeru, welcome to Information Security. Can you add some more information, such as which operating system, and other context? As it stands now it will probably be closed for being to vague.
    – AviD
    Aug 18, 2012 at 20:38

1 Answer 1


Hooks are implemented in a whole bunch of ways:

  • Modifying legitimate jump instructions to point at hooks instead of the normal code.
  • User call table (IAT) hooking - modifying the addresses of user-mode APIs in a process.
  • Kernel call table hooking (e.g. SSDT / GDT ) - replacing a call table pointer with the address of your hook.
  • WndProc hooks (e.g. PeekMessage) - hooking onto window notification messages.
  • Legitimate callbacks like PsSetCreateProcessNotifyRoutine.

I'm guessing you're most interested in the first two types.

Jump hooks can be created in a near-infinite number of ways. This makes it almost impossible to write a tool to identify the hooks. However, you can use integrity checking tricks, e.g. comparing code in the binary file (e.g. exe or dll) to the code in memory. You could also hook WriteProcessMemory and other such APIs to detect modification of process memory, though this only works against user-mode attacks.

IAT hooks are a little easier to check for. Take a snapshot of the IAT of a process when it starts (e.g. from the static binary) and compare the in-memory IAT to the real addresses of the functions that should be in there. For example, if you know IAT entry 4 points to user32.MessageBoxA, you can use GetProcAddress to find the real address of that function and compare the address in the IAT to that. If they don't match, you know it's been hooked.

For further reading, there's a great paper on the subject, and I highly recommend reading "The Rootkit Arsenal" by Bill Blunden.

  • 1
    Not much to add - except that PatchGuard on 64-bit Windows already checks for SSDT/GDT modifications, so there's no real need to do that bit yourself. However, legitimate hooks exist in the form of filter drivers (often used by AV vendors) whose detection and removal is doable but might cause problems!
    – user2213
    Jul 31, 2012 at 11:44
  • @Ninefingers Thanks for the added info. There are actually ways to disable the PatchGuard checks, they're covered in The Rootkit Arsenal.
    – Polynomial
    Jul 31, 2012 at 11:47
  • @HeeruWest Integrity checks are your only solution then. Jump hooks are notoriously difficult to catch.
    – Polynomial
    Jul 31, 2012 at 11:48
  • 1
    @Polynomial true, but given Patchguard's been bumped several times and works in different ways in each version, I'm not sure how reliable disabling it would be for malware. But yes, you could, if you so wished; in which case if you'd written a malware detection tool you'd have to hope the author didn't decide to disable that as well (some time ago malware did target & disable certain AV products).
    – user2213
    Jul 31, 2012 at 11:51
  • 2
    @Ninefingers I completely agree. Once the malware is in the kernel, you're pwned anyway. Nothing can be trusted. I was just stating that there are ways around PatchGuard, so it shouldn't be relied upon for security.
    – Polynomial
    Jul 31, 2012 at 12:05

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .