I once heard the NX bit was a panacea, then that it was not. One detail I've wondered about though: Does the NX (no execute) bit protect against code inserted into the stack and executed there? It seems to me that the stack, because it's not the heap, might not be protected by NX usually. THanks.
The NX bit is a feature of the Memory Management Unit of some CPU (including recent enough x86). It allows to mark each memory page as being "allowed" or "disallowed" for code execution. The MMU is under control of the kernel; the kernel code decides which pages get the execution privilege and which do not. Therefore, whether the stack space is protected against execution depends on the OS. It also depends on a lot of other things:
It so happens that in the ELF file format, which is the format of executable files and DLL on Linux, there is a field which allows to specify whether the stack should be executable or not. When GCC compiles some code which contains a nested function, it sets this flag in the produced binary.
I have made some tests on a fairly recent Linux (Ubuntu 13.10, on 64-bit x86, with NX bit activated). Whether a given page is executable or not can be inferred from the contents of the special file
From all of this we conclude that on at least some recent versions of Linux, the stacks (main and threads) will usually be marked as non-executable (i.e. NX bit set). However, if the executable code of the application, or some executable code somewhere in a DLL loaded by the application, contains a nested function or otherwise advertises a need for an executable stack, then all stacks in the application will be marked as executable (i.e. NX bit not set). In other words, a single loadable plugin or extension for an application may deactivate NX stack protection for all threads of the applications, simply by using a rarely used but legitimate and documented feature of GCC.
There is no need to panic, though, because, as @tylerl points out, protection afforded by the NX bit is not that great. It will make some exploits more awkward for the least competent of attackers; but good attackers will not be impeded. Also, all this talk about NX is about trying to contain damage once a buffer overflow or use-after-free has occurred, and that's arguably a bit late to react.
By marking the stack as non-execute, you effectively prevent code inserted into the stack from running. You're not protecting the stack from modification; rather, you're causing a hard crash when the code attempts to jump to a position in the NX-marked stack.
The workaround is to not attempt to execute code on the stack. Instead of setting the return position to a spot in the stack, you set it to the location of a system function such as
NX therefore provides some level of protection, but not tons. It somewhat limits the flexibility of what an attacker can do with the initial exploit, but given he can leverage that exploit to execute a shell, there's not a lot of safety there.
The next layer in defense is ASLR, which randomizes the location of system functions such as
Attacks against ALSR are typically brute force; run your exploit multiple times, trying different return addresses each time. This means that ALSR is significantly less effective on 32-bit system than on 64-bit because of the limited range of randomization, allowing for relatively fast brute-force attacks.
ASLR combined with NX offers some degree of protection, but nothing is absolute.