Take the 2-minute tour ×
Information Security Stack Exchange is a question and answer site for Information security professionals. It's 100% free, no registration required.

Many malware executables use hooking to be invisible. I am familiar with some hooking methods, I'm not sure what methods exist for detecting them. Is there a way to tell when a function call has been modified to perform other tasks through hooks?

share|improve this question
    
Heeru, welcome to Information Security. Can you add some more information, such as which operating system, and other context? As it stands now it will probably be closed for being to vague. –  AviD Aug 18 '12 at 20:38
add comment

1 Answer

Hooks are implemented in a whole bunch of ways:

  • Modifying legitimate jump instructions to point at hooks instead of the normal code.
  • User call table (IAT) hooking - modifying the addresses of user-mode APIs in a process.
  • Kernel call table hooking (e.g. SSDT / GDT ) - replacing a call table pointer with the address of your hook.
  • WndProc hooks (e.g. PeekMessage) - hooking onto window notification messages.
  • Legitimate callbacks like PsSetCreateProcessNotifyRoutine.

I'm guessing you're most interested in the first two types.

Jump hooks can be created in a near-infinite number of ways. This makes it almost impossible to write a tool to identify the hooks. However, you can use integrity checking tricks, e.g. comparing code in the binary file (e.g. exe or dll) to the code in memory. You could also hook WriteProcessMemory and other such APIs to detect modification of process memory, though this only works against user-mode attacks.

IAT hooks are a little easier to check for. Take a snapshot of the IAT of a process when it starts (e.g. from the static binary) and compare the in-memory IAT to the real addresses of the functions that should be in there. For example, if you know IAT entry 4 points to user32.MessageBoxA, you can use GetProcAddress to find the real address of that function and compare the address in the IAT to that. If they don't match, you know it's been hooked.

For further reading, there's a great paper on the subject, and I highly recommend reading "The Rootkit Arsenal" by Bill Blunden.

share|improve this answer
1  
Not much to add - except that PatchGuard on 64-bit Windows already checks for SSDT/GDT modifications, so there's no real need to do that bit yourself. However, legitimate hooks exist in the form of filter drivers (often used by AV vendors) whose detection and removal is doable but might cause problems! –  user2213 Jul 31 '12 at 11:44
    
actually my problem is on the 1st one.thanks –  Heeru West Jul 31 '12 at 11:47
    
@Ninefingers Thanks for the added info. There are actually ways to disable the PatchGuard checks, they're covered in The Rootkit Arsenal. –  Polynomial Jul 31 '12 at 11:47
1  
@Polynomial true, but given Patchguard's been bumped several times and works in different ways in each version, I'm not sure how reliable disabling it would be for malware. But yes, you could, if you so wished; in which case if you'd written a malware detection tool you'd have to hope the author didn't decide to disable that as well (some time ago malware did target & disable certain AV products). –  user2213 Jul 31 '12 at 11:51
2  
@Ninefingers I completely agree. Once the malware is in the kernel, you're pwned anyway. Nothing can be trusted. I was just stating that there are ways around PatchGuard, so it shouldn't be relied upon for security. –  Polynomial Jul 31 '12 at 12:05
show 7 more comments

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.