First, some background.
We're going to be digging into the guts of the Windows security model a bit here.
Processes are securable objects (much like files, registry keys, devices, shared memory sections, named pipes, etc.). That means they have a Security Descriptor, which contains an ACL (Access Control List) and other details like their Owner. In the case of processes, the security descriptor can be set by the parent process when calling a CreateProcess-family function, but usually nobody bothers and it's just set to some default. The default it allows the owner (that is, the user it runs as) and the SYSTEM user to have full control, and everybody else gets basically no privileges. You can see this in the Process Explorer tool from Sysinternals; select a process, go to Properties, go to the Security tab, and click on Permissions (at the bottom).
This ACL is checked when a process calls a function in the OpenProcess family. Much like CreateFile or RegOpenKey, these functions require the caller to specify the permissions that they want to have on the target process. The OS then checks those permissions vs. the ACL of the target process to see whether the security token (which identifies the owning "user" or "account" or similar, plus Groups and any special Security Identifiers "SIDs") that the calling process runs as are granted (and not also denied) the requested permissions on the callee process. This is exactly the same as when a process tries to open a file or similar, except instead of checking a file's ACL, it's checking a process' ACL.
You can read more about process security in Windows at Process Security and Access Rights
The (probable) answer to your question
So how does a process prevent itself from being opened by other processes (as is required by Process Explorer to get the process details, or by a DLL-injecting process trying to get access to the target process' memory)? Simply call SetSecurityInfo, passing a HANDLE to itself as the first parameter (there's actually a special HANDLE reserved to indicate "the current process, from the perspective of the caller"; it always has full access and doesn't require a call to OpenProcess first; this value is returned by GetCurrentProcess). In the other parameters to SetSecurityInfo, specify that you are setting the DACL (discretionary ACL, the kind that controls who has permissions to access the object in various ways), and pass an empty DACL (one that grants no privileges at all), or a similarly limited DACL. (Note: do not pass NULL for the DACL; that removes the DACL and allows everybody unrestricted access.) You can do this very early on in the main method of your program, such that anybody who tries to open this program's process object will get Access Denied.
Wow, is it really that simple?
Eh, not really. There's a bunch of stuff going on here that you need to consider if you want to do this yourself, or that is relevant to you if you want to bypass it.
- It's possible to open an object (such as a process) without specifying any requested permissions at all. In that case, so long as the object is at least visible to the caller, there are no permissions to check against the target's DACL, so the call succeeds and you get back a HANDLE or similar. Of course, it has no granted permissions, which is mostly useless.
- ACLs can be overridden. Obviously, anybody who has been granted the relevant access can modify a process' ACL, but also, any process running as the owner of the process can always override the ACL even if the owner isn't actually permitted by the DACL to modify anything. This doesn't happen automatically, though, which is why other processes running as the same user will still get
Access Denied errors when they try to open the process normally; they first must open it without requesting any privileges, and then overwrite the ACL using "permissionless" HANDLE.
- With sufficiently high kernel privileges (a concept equivalent to Linux capabilities) - specifically,
SeTakeOwnershipPrivilege (in practice, this mostly only is assigned to processes running as Administrators or SYSTEM) - a process can override the owner of an object (such as a process). This doesn't change who the process is running as, but it does change who has the "set the ACL regardless of privileges" permission from the bullet point above. This is also not automatic; a process must first open the target with no privileges specified, then specifically request to take ownership, then request to overwrite the ACL. It can then re-open the target with whatever privileges it granted itself. This of course means it can take over anything, and then give itself whatever privileges, so of course it's tantamount to complete ownership of the OS.
- Kernel privileges can do a lot! There are also a few that let you just ignore specific aspects of the ACL, rather than overwriting it (although overwriting it could be viewed as ignoring the
WRITE_DAC permission). For files specifically, there's the SeRestorePrivilege, which allows (over)writing files without regard to their DACL, but I don't think it works on processes. However, SeDebugPrivilege does work on processes, and does what it says in the name; a holder of that privilege can control processes in the way a debugger needs to, including reading and writing process memory and suspending or resuming threads, regardless of the process' DACL. Yes, this also de facto means complete control of the operating system. Anyhow, this privilege is why processes running as Administrator are able to access the process details.
- There's a window of vulnerability here. While a parent process could always create the child process with such a "nobody can access this" DACL, if the child process wants to set its own DACL, it can't do that until the process starts executing. However, that creates a (short) window at process initialization when an external process could open a HANDLE with meaningful access to the child process because during that time, the child process still has the default security descriptor. This is especially relevant if the child process was created suspended, but could happen even without that if the "attacker" process is responds immediately when a new process starts (I believe you can register an event handler for whenever a process starts).
- There's another vulnerability here: the parent process. A parent process always receives a HANDLE to the child process (it's in the
lpProcessInformation out-parameter) and that HANDLE has full privileges. Even if the parent never set the child's security descriptor, and even if the child modifies its DACL before the parent ever does anything with the returned HANDLEs, the parent can still use the child process' HANDLE to do whatever the parent wants. The permissions a HANDLE grants its holder are set when the HANDLE is created, and are only checked against the target's security descriptor (and caller's security token) at creation time. After that, so long as the target exists and the HANDLE is never closed, it can be used regardless of any changes to the target's security descriptor.
