I'm trying to hook some functions in a static linked executable and I have to get a pattern for the function in almost every new version of the program as the function is quite big so the compiler changes many of the bytes for every little change.

I noticed that this function uses another very small function that is also statically linked but I can get a pattern for it which works for almost all the versions.

So I'm planing to search for this function, hook it, trace the stack and finally find the caller function address.

I didn't know what is exactly the stack of a function and thought CaptureStackBackTrace returns the addresses of the functions but this was wrong. It returned some addresses that are near to the real functions addresses.

Can I get the functions start addresses from these addresses returned from CaptureStackBackTrace? I don't need to use dbghelp to get the names or addresses of functions as pdb files won't be presented.


A year old question but it does come up in searches so:
Pattern matching:
First part of your question, you might want to look up on how to find functions by signatures/patterns. Web search for this, there are a lot of good answers on this or the sister site Stack Overflow, on Game hacking sites, etc.
See: https://www.unknowncheats.me/forum/c-and-c-/125497-findpattern-benchmark.html

For the stack trace part:
CaptureStackBackTrace() works by calling RtlWalkFrameChain() walking EBP/RBP stack frames. Disassembling the function in ntdll you can see it's a very simple function. At least one problem of it is for functions that don't have stack frames, it's probably not going to find those return addresses. This and StackWalk64() are better left for debugging situations where you have PDBs for everything, etc.

What you'll probably want to do is walk up the stack yourself. You can get the current stack pointer (ESP/RSP) using the _AddressOfReturnAddress() compiler intrinsic or do a "__asm mov eax,esp", etc.

Using "DWORD_PTR" advance up the stack calling VirtualQueryEx() on each value that looks like an address. If it returns success it's at a least a valid memory pointer. If it's protection flag is "(PAGE_EXECUTE | PAGE_EXECUTE_READ | PAGE_EXECUTE_READWRITE | PAGE_EXECUTE_WRITECOPY) != 0" then you know it's not executable.

If the type is not "MEM_IMAGE" then it's probably not a return address either because all normal Windows executables (.exe, .com, .dll, etc.) are going to be imaged mapped. With some exceptions like if the code is from some sort of JIT, shellcode, mallware, etc.

Then you can go a level or two deeper by using a simple dissembler library (like HDE) to disassemble the instruction at the assumed return address. Should be a valid instruction there.

Look at the "debghelp.dll" functions too. There's support to resolve addresses to known API functions, etc.
Update: Tried this. Depending on the target process it ends up being a mess since you will find a bunch of stray code offsets. Turns out the stack is relatively chaotic. Better to use CaptureStackBackTrace() and hope that the return chain has frame pointers.

For the get function start address question:
(A lot of these topics are not so trivial) you can try using "SymFromAddr()" and that might tell you what API the return address is in.

If you can use the dbghelp.dll functions or they don't work for you, if again a return address has a MEM_IMAGE type the "AllocationBase" should the same as "HMODULE". The module that contains the function returned too.

Now that you found the base of the module you are in:
Then using PE header parsing, you can walk the export table for the module. Loop through them comparing the start address of one the exported function to the next. Your address will be between there someplace. The exported function closest to your address is the answer.
Update: SymFromAddr() IS what you want to use. Works great with the stack walking described above. Be sure to wrap it around a critical section though, it's not thread safe.

Note: I find my self with Windows API functions they don't always work as they are described and/or don't handle edge cases so well where you pretty much have to experiment; using either kernel32.dll, psapi.dll, or even ntdll.dll functions directly.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.