Given extensive protections in modern operating systems that make buffer overflow exploits unfeasible, should I even bother studying these?

Question

I’ve been diving into the world of buffer overflow vulnerabilities and their exploitation, which has been both challenging and fascinating. However, I’ve recently hit a mental roadblock and would love to get your insights.

With modern operating systems like Windows 11, numerous protections (e.g., ASLR, DEP, Control Flow Guard) make exploiting buffer overflows seemingly impossible under normal conditions. From what I’ve seen, successful exploitation often requires explicitly disabling these protections.

This raises a question:

If exploitation only becomes feasible when protections are deliberately disabled, doesn’t this shift the issue from being a vulnerability to a misconfiguration problem? In such scenarios, how relevant is studying buffer overflow attacks today, especially when modern systems seem almost invulnerable unless improperly configured?

I understand that the concepts are foundational to cybersecurity and have broader applications (e.g., understanding exploit chains or legacy systems), but I can’t shake the feeling that this area is becoming less practical in modern environments.

So, is it worth continuing to invest time in studying and practicing these techniques?

Answer 1

If exploitation only becomes feasible when protections are deliberately disabled, doesn’t this shift the issue from being a vulnerability to a misconfiguration problem?

Security misconfiguration is still a security issue that can lead to a vulnerability, so from a security point of view that is still relevant.

In addition, while making buffer overflow exploits more difficult, I think it's wrong to say that buffer overflow exploits are only feasible when protections are disabled. For example, there are ways of getting around some of these defences

This blog post about an OpenSSL buffer overflow vulnerability from a couple of years ago is illustrative of the current state of affairs. The CVE was initially release as CRITICAL because "it is an arbitrary 4-byte stack buffer overflow, and such vulnerabilities may lead to remote code execution" but was later downgraded, partly because of modern stack overflow protections. However, it was still rated as HIGH because:

we have no way of knowing how every platform and compiler combination has arranged the buffers on the stack and therefore remote code execution may still be possible on some platforms.

So buffer overflows are still clearly an issue, but becoming less so over time due in part to modern stack overflow protections. This trend is clear from Mitre's top 25, where "Improper Restriction of Operations within the Bounds of a Memory Buffer" has dropped from number 1 in 2019 to number 20 in 2024. This trend is likely to continue along with a switch to memory safe programming languages for code traditionally written in C / C++.

is it worth continuing to invest time in studying and practicing these techniques?

That depends on your interests and goals. For example, if you want to build secure web apps, there are definitely more important risks to focus on. But if you're interested in securing programs written in memory unsafe languages, then buffer overflow exploits are still relevant, though likely to become less so over time.

Answer 2

In addition to the other answers, it's probably worth mentioning that the protections you quote are mostly there to prevent injection of code to execute (e.g. I send data that exceed the buffer, the buffer is stored on the stack so it changes the contents of the stack to include code and to "return" to that code to execute it).

But while directly injecting code may become more difficult, it does not mean other forms of buffer overflows can't happen and can't have consequences.

Imagine the following (very hypothetical, probably not even valid C) code:

typedef struct
{
   uint8_t buffer[128];
   uint32_t mode;
} thing_t;

process_data(uint8_t *received_buffer, size_t received_len)
{
    thing_t thing;

    thing.mode = 0;
    memcpy(thing.buffer, received_buffer, received_len);

    other_function(thing);
}

If I send data that is longer than the 128-byte limit, I will clobber mode. And the function I call with that struct will now use the mode I sent it, not the one that code wanted (mode 0).

So while you can't just make the target execute any arbitrary code like you would in some attacks, you can definitely make it do things different from what was intended, and this can have lots of consequences, depending on what you can overwrite:

• data could be corrupted
• authentication could be bypassed
• the execution flow could be changed etc.

While there are protections that can be implemented in the processor or in the kernel, they are necessarily limited: neither knows the actual structure of the data you are manipulating, and it can't prevent your code from accessing its own data any way it wants. How could a processor or kernel make a difference between copying data into thing.buffer and making a copy of thing (which legitimately will include both buffer and mode)?

The language itself, and the compiler that go with it, can know the difference and can prevent such issues, but the CPU or kernel can't. Applying "general" rules likes "don't execute code on the stack" or "don't execute code in the data sections or heap" are simple and easy, but they only address a small subset of buffer overflows.

So while it may change the exact type of buffer overflow attacks one can perform, it certainly doesn't prevent all of them.

Another important point is that while some operating systems and CPUs can integrate such protections (often the OS needs the CPU to implement those features), it does not mean that all do:

• There are lots of systems with older kernels or CPUs with no support for those features
• There can be many mis-configured systems
• Some systems have very specific operating systems, notably network equipment (routers, load balancers...), and they often rely on code written non-memory-safe languages. Same thing for embedded systems (you know, all that "IoT" stuff).

Answer 3

They are more rare, but there are still sometimes reasons to bypass overflow protections

You don't see it very much in business infrastructures, but when talking about things like videogames, physics models, etc. It is not uncommon for a developer bypass protections in an effort to improve performance.

ASLR, DEP, Control Flow Guard, etc can prevent certain kinds of stack overflow attacks by creating non-executable memory segments and randomized memory spacing between data objects, but it does not prevent developers from cramming multiple pieces of data into an uninterrupted chunk of memory.

For example, if you need to track millions of objects, traditional OOP data structures may be way too bloated for your use case; so, you can make your system several times faster by dumping everything into a single long variable. It's way harder to secure, way harder to refactor, way harder to maintain stability, etc... but sometimes all of those trade offs are worth the difference of being able to play a game at 3fps vs 30fps.

Voxel games (like Minecraft, Space Engineers, etc) are a big one. When you use something like Unity or Unreal Engine objects to store individual block information, each block contains long unique index ids, canaries, position, velocity, and rotation data, etc. each block could easily take up over a kilobyte of memory, and each time you change one, it has to pass through the engine's overflow protection and that is before you even hit any of the OS level protections.

One solution I've seen to this is to break up your grids into chunks like a 16x16x16 subgrid of blocks, and store them in one binary string where each block is a fixed 13 byte string containing only the block's type and HP, then the grid and/or subgrid would contain all of the position, indexing, rotational data, etc . This way, a fixed 54kb binary string could contain a subgrid of anywhere from 1 to 4096 blocks in memory instead of ~1kb per block. If the average subgrid is 25% full, it means that the game could load on average ~10x as many blocks into memory as a similar voxel game not using such subgrids.

Furthermore, a developer can typically get away with skipping checks on internally generated data. So, if the server accepts a packet from the client that makes the claim to the effect that "I hit X block for Y damage with Z weapon", then this could trigger explosion mechanics where you might need to damage or destroy hundreds or even thousands of blocks. If you let your game engine do it's native overflow protection, it means that every block caught in the blast needs to be checked for overflow, and it needs to have all those spacer memory blocks defined by ASLR, DEP, etc. Instead you can make your own function that checks first that X,Y, and Z are all valid inputs, and then you feed them into your damage propagation algorithm. If you check that the total damage you receive does not overflow, and the explosion function can only spread out that valid damage amount into smaller numbers then there is no point in checking for overflow on each block when applying damage because A >= A/B when B >= 1. So, when you go to rewrite your subgrid, you have no reason to check for overflows because you already prevented them upstream. So, this can save your system from tons of unnecessary overflow checks and gaps/jumps in your stack every time something explodes.

Answer 4

Other answers have described scenarios when these protections are deliberately disabled, but I think there's another important angle to answer here. The main premise of your question - that modern protections make exploitation infeasible - is also flawed. In fact, ASLR, DEP and CFG can all be bypassed.

Based on how you asked your question, I believe you're familiar with basic stack buffer overflow attacks, where you write a good amount of arbitrary data out of bounds, clobber a return pointer sitting on the stack, then pivot execution to a payload that you also write to the stack.

ASLR

ASLR makes a basic buffer overflow difficult, because you can't find your own stack address. Without knowing your own stack address, you can't overwrite the return pointer to point there.

In a real exploit, you might start looking for a memory disclosure bug - something that allows you to leak pointers from your target's memory space. While ASLR can randomise the base address of the stack, it doesn't randomise the offsets between stack objects. Functions typically have fixed-size stack frames, and you can usually plot a predictable path between functions of interest in a target. This means that if you can leak any stack pointer from anywhere in your target, you can figure out the precise addresses of every object that might exist in the stack.

This requires a separate primitive from the out of bounds write in your buffer overflow, which definitely makes the attack harder, but it definitely doesn't make it infeasible.

DEP

Marking the stack and heap as non-executable also stops your basic buffer-overflow from working. While the overflow means you can still clobber a return pointer, you can't execute your favourite payload because you can't put it anywhere executable.

The standard method for working around this is called Return-Oriented Programming, or ROPping. Your target binary has to be mapped into an executable region of memory, so you know you can execute any code still in there.

Return-oriented programming involves finding small sequences of instructions called "gadgets" which give you useful primitives. One gadget might pop a value from the stack into rcx. Another gadget might read something from the memory address stored in rcx into rax. Another might jump execution to the address stored in rcx. ROPping involves building a chain of these primitive gadgets to do something more complex and useful.

If you don't have anywhere that's both writeable and executable, a typical ROP chain might set up a call to VirtualAlloc, to map in a region of memory that's both writeable and executable, then a call to memcpy to copy a payload from the stack into the memory you got out of VirtualAlloc, then jump execution into that payload.

ROP chains are much fiddlier than just uploading a payload and jumping to it, but any reasonably complex program should give you enough gadgets to do what you need to.

ASLR will randomise the addresses of these gadgets, making life a little harder. Again, your first step would probably be to find a memory disclosure vulnerability. If all your gadgets live within the same .exe or .dll file, then ASLR will only randomise that file's base address in memory; the relative offsets of everything in that file will remain static.

This, again, means that a single pointer disclosure will be sufficient to learn the address of every gadget in your ROP chain, giving you a feasible attack.

CFG

CFG is Microsoft's name for their Control-flow Integrity Checking technique. In general, control-flow integrity checking ensures that a code pointer always points to some "safe" place. Whenever your target makes a jump based on address in memory, it first checks that that address is a sensible place for it to jump to. Given that a ROP chain often involves jumping part way down a function, perfect CFI makes this impossible.

I've not needed to think much about control flow integrity checking myself, but other answers on this site note that perfect CFI basically doesn't exist.

Sometimes the checks are quite coarse - checking merely that jumps are heading into the right page of memory rather than to the correct instruction within that page. The Wikipedia article on CFG notes that it can be bypassed by jumping into a module which doesn't use CFG, which suggests that's the case there.

When they are fine-grained, they often rely on hidden information that can then be leaked. For instance, PaX RAP XORs each functions return pointer with an encryption key each time it's called. The key needs to be fixed through the lifetime of several function calls, and is stored in one CPU register. The encrypted return pointer lives in another register. On return, the function then decrypts the encrypted return pointer and compares it to the address it's about to jump to. If it's different, it crashes.

Armed with some good memory disclosure, though, it's possible to derive this key. Because the encrypted return pointer lives in a register but changes on each function call, the encrypted pointer needs to be pushed to the stack between function calls. If you've already defeated ASLR, and can leak an encrypted return pointer, you know what its value should be, so you can figure out what the encryption key is. This means that, rather than targeting the current stack frame with your buffer overflow, you can target the previous frame and clobber both the return pointer and the encrypted return pointer.

Yes it's worth learning

Yes, of course, it's still worth learning about basic buffer overflows. You might have noticed that the techniques up there are much harder and make more assumptions, but they build directly from the basic techniques you now already understand. In my view, there's no way to understand how to bypass ASLR unless you've got a good understanding of the attack class that it mitigates

This also means, on the flip-side, that it is never sufficient to just turn on all the "make my code secure please" compiler flags and assume you're now invulnerable. All these techniques make exploitation harder, but with enough effort and sufficiently good bugs, it's nowhere near impossible