My lecturer posted this in his slides and I have no idea what he is trying to say.

I quote from his slide:

Windows DEP = Data Execution Prevention

Old “X” idea: must allow explicitly, current OS+programs would stop working.

The “NX” idea: Never Xecute = can forbid. Easier to make compatible systems.

Please do not provide technical details such as what DEP is trying to achieve, dramas with keeping APIs compatible.

I am confused on the old vs new point he is trying to make, especially in terms of compatibility.


I'm not 100% sure what point your lecturer is trying to get at, particularly because "DEP" is a vague term from a historical perspective.

I know that you asked us not to provide technical details, but I think they're warranted here from a history perpective.

There are two types of DEP:

  • Software-enforced DEP, which is actually SafeSEH (nothing to do with preventing execution of non-executable memory pages)
  • Hardware-enforced DEP, which allows the OS to mark memory pages as non-executable in such a way that the CPU will refuse to execute code in them.

You might be wondering "why the heck did they call both of these DEP?", and the answer is essentially that back in the early 2000s SEH pointer overwrites were the primary trick used to exploit stack buffer overflows, so "software data execution prevention" was a reasonable name for an exploit mitigation that prevented an attacker from redirecting EIP using overwritten stack data. We now call this SafeSEH instead.

The second half of the equation in such stack buffer exploits was to redirect EIP to a buffer that the attacker controlled (in this case the stack) which is where hardware-enforced DEP comes in. By utilising the NX feature (Intel calls it XD, but NX caught on as the common term) of the CPU, the OS can mark pages as non-executable and if an attacker tries to point EIP to some memory inside that page the CPU will refuse to fetch instructions from it and raise a hardware exception. As part of enabling DEP, the OS sets the stack up to be non-executable memory when creating the process. As this was the second part of preventing exploitation via stack buffer overflows, and it was enabled by the hardware NX bit (also called the XD bit) it got the name "hardware DEP".

(intermission) If you're curious about SafeSEH, it works by maintaining a list of valid exception handlers at compile time and the exception dispatcher won't transfer execution to a location not on that list. There's also SEHOP, which walks the full exception chain before dispatching to ensure that it is intact and complete (there's a special SEHOP record injected onto the end of the chain so it can find the end) - this prevents an exploit from overwriting a handler entry. The bonus of SEHOP is that you don't have to recompile the application with SafeSEH support.

From now on I'm just going to use the modern terms (DEP and SafeSEH) because otherwise this gets confusing.

The way the DEP policy works on Windows is as follows. There are four system-wide policy options: off, opt-in, opt-out, and forced. Portable Executable files (e.g. EXEs and DLLs) can set the IMAGE_DLLCHARACTERISTICS_NX_COMPAT flag in the DllCharacteristics field of the PE Optional header to indicate that they are NX-compatible (i.e. they know about NX and won't try to execute non-executable pages). The combinations are as follows:

  • When the DEP policy is set to AlwaysOff, the NX_COMPAT flag is ignored and DEP is not enabled for any process.
  • When the DEP policy is set to OptIn, the OS honours the NX_COMPAT flag; processes without it don't get DEP, processes with it do get DEP. However, the administrator may set up Image Execution Options for particular applications to force DEP to be on for them regardless of the value set in the PE header. Additionally, the process can call the SetProcessMitigationPolicy API with ProcessDEPPolicy to enable DEP on itself (although this is unlikely, as processes that are DEP-capable would likely just set the PE flag). This API can also be used to permanently enable DEP for the process (i.e. a later call to the API telling it to disable DEP would be denied).
  • When the DEP policy is set to OptOut, DEP is enabled for all processes by default, regardless of the NX_COMPAT flag. Like with the OptIn policy, the administrator can configure Image Execution Options for particular applications to disable DEP. Again the SetProcessMitigationPolicy can be called by the process with ProcessDEPPolicy to turn DEP on or off, or permanently enable it.
  • When the DEP policy is set to AlwaysOn, the NX_COMPAT flag and Image Execution Options are ignored and DEP is enabled for every process. The SetProcessMitigationPolicy API cannot be used to disable DEP when this policy is set; it is permanently enabled for all processes.

With all this in mind, there are a few things I think your lecturer might be trying to say:

  • Originally when NX came out, OS devs thought about setting everything to non-executable and making applications mark things as executable, but this would lead to problems with applications that incorrectly assumed that non-executable pages could be executed. This would mostly affect packed/compressed applications and anything with a JIT engine. So, instead, OS devs decided to default to letting processes just execute any data they like unless they opted into DEP, for compatibility reasons. In this case "X" is the original thinking, and "NX" is the approach they ended up taking.
  • SafeSEH (previously known as "software DEP") requires applications to be recompiled with the feature enabled, whereas SEHOP can just work on application that aren't compiled with SafeSEH. In this case SafeSEH is "X" and SEHOP is "NX".
  • Originally we were just going to have a policy of having applications be DEP by default but this doesn't work in practice, so we built a load of policy options (OptIn, OptOut, AlwaysOn, AlwaysOff). In this case "X" is the original on/off approach, and "NX" is the policy approach.

Sorry that I couldn't give you a simple straight answer, but hopefully this has somewhat improved your understanding of how the terminology evolved and how the current NX/XD/DEP/SafeSEH/SEHOP policies and ecosystem came together.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.