I have read that the PE loader is responsible for loading executable images from disk. When and where is the control flow exactly transferred to the loader? The PE format is well documented but there seems to be a little info regarding the functioning of the loader itself.
To answer your question, I need to cover the full description of how a new process created.
I'll paraphrase an overview of what's said in the book, as copying it verbatim would probably have some copyright issues.
The PE loader is exposed by a set of user APIs in
kernel32.dll, under the
CreateProcess family. There are different APIs for doing different things, e.g. running a process under an alternative security context.
Here's how it works:
- The user-mode API validates the input parameters, and converts them to their system (native) counterparts.
- It then opens the executable file and loads it into memory.
- Creates the executive process object in the kernel. This involves populating the
EPROCESSstructure and registering the process in various lists.
- Creates the main thread of the process (stack, execution context, thread object)
- Performs subsystem-specific initialisation - e.g. CLR init for .NET applications.
- Start the main thread (unless it was created with the suspended flag set)
- Load the appropriate DLLs in the context of the process.
Most of this is done at the kernel level, using the appropriate
Ps-prefixed native functions. The full set of steps involved is rather complex (in fact, it takes up 15 full pages in the book) and involves a lot of different actions depending on the susbsystem used.
The tricky part with your question is that the "loader" isn't really something that gets control flow. The instant you call
CreateProcess, you're technically running the loader. However, the kernel part of the loader begins when
ntdll!NtCreateUserProcess transitions into kernel-mode. If we're really strict about it, we might say that the first part of the loader is
PspAllocateProcess, since that's what allocates the initial structures.
In the book Mastering Malware Analysis: The complete malware analyst's guide to combating malicious software, APT, cybercrime, and IoT attacks [Alexey Kleymenov, Amr Thabet], there're 2 sections in chapter 2 called "Process loading step by step" and "PE file loading step by step" which document how the Windows PE loader is loaded and how it works.