Is using separate stacks for return addresses and function arguments a viable security measure?

Question

As far as I know, many exploits rely on overwriting return address of the function they try to exploit. They do it by buffer overruns. But what if the compiler set up two separate stacks far from each other in the address space, and used one of them (maybe the system one, like esp/rsp-driven one on x86* systems) for return addresses and the other to pass function arguments? In this case any buffer overruns will overwrite some locals, maybe including those of caller functions, but still will leave the return addresses intact.

Also, the stack for return addresses could be used to store all the function pointers and similar entities which allow to indirectly jump/call, so that they would also be protected.

Has this option been already considered? Is it viable? If yes, why hasn't it been implemented (or has it?)?

Answer 1

The problem with most operating systems is that they follow a specific "calling convention." This convention requires putting function parameters on the stack, being some derivative of the C-style calling convention. You must use this convention for ABI (Application Binary Interface) compatibility with that OS. So, without OS support, you could only use this feature for calls made within your application.

This would complicate compilers quite a bit and probably require a fair amount of work. In short, you could protect your own programs if you had a compiler that supported this calling convention, but you'd still be at the mercy of the OS whenever you had to do things like reading/writing a file, etc. A buffer overrun in a DLL, for example, can't be fixed by you changing your calling convention.

Secondly, until recently, with the advent of virtualization, it really wasn't feasible to set up a separate area like this, because segmentation was expensive, and memory virtualization even more so. Today, this would practically be a non-issue, but since we have to deal with historical software (e.g. stuff written ten years ago that still require the conventional calling methods), the OS would then be forced to support both models for some indefinite period of time.

If a new OS with no compatibility concerns were written, it could certainly do this, but it probably won't happen, because there are more viable methods. Microsoft's own Singularity OS is completely immune to buffer overruns (according to them), because the OS statically validates that each program cannot possibly misbehave. Interestingly, this OS uses no "memory protection" as used by Windows, Linux, Mac OS, etc. The programs are validated for correct behavior before they run, not as they run. Of course, if a virus were capable for this system, it would have unlimited system control because of the lack of protection at the hardware level.

In short, even without any serious research on the topic, it's clearly possible that this approach would work, but outside of FOSS (Free and Open Source Software), it wouldn't be possible for this approach to work from a financial standpoint. Linux could be re-written from the kernel up to support the new model, a compiler rolled out for it, and then all the software out there could be re-compiled without too much effort (note: "too much" being relative to, say, Microsoft).

Microsoft, Apple, and so on do not have this benefit, because the code is already compiled by millions of different developers, so anything that couldn't be updated would be instantly obsolete, or they'd have to write emulators to support the old software. Basically, until someone comes up with an OS that has this feature built-in, with compatible compilers (at minimum C and C++, plus probably Cocoa and Win32 C++), and it gains enough support to become a contender against Linux, Microsoft, and Mac OS, it'd be pretty hard to justify moving to a new model. Linux would be the easiest to move over, although all software would have to be compiled until RPMs and other package types supported the new calling model.

Finally, DEP (Data Execution Prevention) pretty much solves the problem in most cases, making it harder to execute code that shouldn't. This also reduces the need to switch to a new model, although, as Singularity demonstrates, hardware could be a lot faster if it wasn't constantly forced to protect against programmers' bugs and the exploits they present.

Answer 2

Yes, this has been implemented before. In this blog post, Erin Ptacek briefly mentions how AVR has different program and data memory and how this makes exploitation more difficult.

A Harvard Architecture has two distinct memories; there is program memory (imem, typically flash) and data memory (dmem, typically SRAM). They live in two different address spaces. The CPU reads instructions from imem, but instructions themselves read and write dmem. This is neat because makes exploits harder to write. You can’t simply upload code in a buffer and somehow point the PC at it; that buffer is in dmem, not imem. You’ll see what I mean in a few weeks.

While this would help prevent an attacker from exploiting a buffer overflow it doesn't totally prevent a buffer overflow from being exploitable. Overwriting local variables can go a long way towards gaining control over a process. Phyrfox also made really good points about why this wouldn't jive with the expectations of modern operating systems.