Information Security Stack Exchange is a question and answer site for information security professionals. It only takes a minute to sign up.
Sign up to join this community
DLL hijacking (I believe) is when a dll with a name identical to a system dll is placed higher up in the search directory precedence: like placing a malicious dll named secur32.dll or winsock.dll next to a trusted executable and waiting for an elevated user to run it.
Seems trivial for an antivirus to detect and quarantine such a dll, but what does Windows itself do to prevent this kind of exploit?
It's less trivial than you might think, because there's legitimate reasons to do this (for example, if your program depends on an older version of a library that you expect will also be installed globally, or simply if you have a name collision between unrelated libraries). Changing this long-lived legacy behavior would be a backward-incompatible change of the sort that Windows generally tries pretty hard to avoid. On the other hand, there are preferred official ways to do this (local DLLs or side-by-side assemblies) so it's possible that the legacy behavior will eventually be deprecated.
Relevant link for following this answer: https://learn.microsoft.com/en-us/windows/win32/dlls/dynamic-link-library-search-order. Note that the search order is pretty complicated, but that there are numerous opportunities to search somewhere else before the application's directory. See also https://learn.microsoft.com/en-us/windows/win32/dlls/dynamic-link-library-redirection and https://learn.microsoft.com/en-us/windows/win32/SbsCs/isolated-applications-and-side-by-side-assemblies-portal, which talk about ways an application can override the usual DLL search behavior to load a specific instance of a DLL, or apps can multiple versions of a DLL on the same system but get the right one at runtime.
The main thing Windows does to mitigate this risk is install programs to a non-user-writable location (the various Program Files directories). If an attacker is able to plant a malicious DLL in the program's install directory, they could have replaced the executable itself, rendering DLL planting totally moot. For that matter, a user with such privileges generally could have replaced the system libraries, added a malicious entry to the KnownDLLs registry key, or other methods of obtaining arbitrary control over nearly all processes on the system.
Additionally, Windows by default runs most programs with their working directory being their install directory. On modern Windows versions, even if the working directory isn't the install directory, it's a lower risk because by default, the working directory has been moved very late in the search order (though it can be moved back to its legacy position of "just after the install directory" if desired).
Many system libraries are added to a "KnownDLLs" list, which is stored in the registry. Requesting a DLL whose name is on that list will take precedence over the install (or working) directories. Obviously though, this doesn't work for all DLLs, in particular it doesn't work for app-specific and usually not for vendor-specific DLLs.
Windows offers developers the means to avoid this risk. Aside from hardcoding the DLL path in a LoadLibrary call (discouraged), developers can call SetDefaultDllDirectories to remove the application's installation directory, and/or SetDllDirectory("") (empty string as parameter) to remove the working directory, before loading libraries manually. Finally, the methods listed above (local DLLs or manifests for side-by-side assemblies) can be used to force the OS to load a known good DLL, even if that's just a system DLL or a copy thereof and (importantly) those methods work for DLLs loaded at process creation, before developer-specified functions can be called. Note that using local copies of system DLLs is discouraged for a few reasons: it increases memory requirement for the system, increases install footprint, and means that if the system library gets patched (e.g. to fix a security flaw), you app will still be running the old version until independently patched.
Obviously, requiring developers to take actions like those to avoid a security risk is bad design - it violates the principle of security by default - but for a combination of the legacy reasons and the permissions-based mitigations listed above, this is what we have.
Last of all, Windows does provide some extreme or last-ditch measures for protection. On the automatic side, anti-virus software such as Windows Defender scans DLLs as well as executables, and may flag malicious ones (though you should never rely on this). On the manual side, system administrators can restrict the executable modules - EXEs, DLLs, etc. - that are allowed to load into a process using various methods, such as by file location or digital signature. That requires a lot of customization though, and is rarely used except on severely locked-down machines or in large organizations with many machines that each need only software from a small list.
