Here's a good tutorial and explination on this. Many antiviruses are taking the same step and using malware like functionality in order to combat malware. Pretty neat how everything is transforming.
Several API hooking libraries exist. Typically, they do the following:
Replace the initial part of a defined function code with our own code (also known as trampoline). Upon execution, the function jumps to a hook handler.
Store the original version of the replaced code of the defined function. This is required for the defined function to operate properly.
Restore the replaced part of the defined function.
About API hooking
Windows API hooking is a process allowing to intercept API function calls. This gives you the control over the way operating system or a piece of software behaves. Some of the software solutions that utilize hooks include: antimalware software, application security solutions, security monitoring tools, system utilities, tools for programming, and many others.
API hook types
API hooks can be divided into the following types:
Local hooks: These influence only specific applications.
Global hooks: These affect all system processes.
The type of hook technique for Windows that we cover here belongs to the global type. It affects all processes across all sessions (as opposed to the SetWindowsHooks method, which is limited only to a selected desktop).