Can stack overflow be prevented by pushing return address first?

Question

I feel that this is a really dumb and obvious questions, but I haven't found any answers online. I mean the textbook example of buffer overflow is done by memcpy into a limited size buffer on stack. It works because function call allocates the buffer before pushing the return address. Overflowing the buffer will overwrite the return address. It seems pretty obvious to try to change the order by pushing the return address first and then allocate the stack space. The compiler knows how much local space each function call needs and should be able to pop the stack at function return to access the return address. Seems like this would easily prevent a lot of overflow issues?

Answer 1

The main reasons are that it is inconvenient, results in sub-optimal code, requires prior knowledge of function stack size, limits performance, violates the specification of many calling conventions, and doesn't actually prevent overflows from succeeding.

The usual way that a function calls into another function, at least on x86, is (unsurprisingly) with the call instruction. This instruction pushes the return address onto the stack and redirects the instruction pointer to the target address. The stack is a concept that the processor itself understands and facilitates, and we can't change the processor's behaviour.

In order to facilitate putting the return pointer before stack allocations, we would need to allocate the function's stack space before the call instruction is executed. This requires the calling function (the caller) to allocate the stack for the called function (the callee), and thus inherently requires that the caller clean up the stack of the callee. This is possible in calling conventions such as cdecl (also some cases with thiscall), which utilise caller clean-up, but is a violation of the calling convention specifications for stdcall, fastcall, vectorcall, pascal, and a number of other conventions.

You might be wondering why it matters whether stack allocation is done by the caller or callee. Well, in general, a function exists because it contains code that is used in more than one place (this is why we have inlining optimisations!). In a callee-cleanup model, the code for setting up the function's stack allocation space and cleaning it up at the end is part of the callee function itself. The calling function doesn't need to know about the stack memory requirements of the callee function. This is useful because you don't end up duplicating the stack management logic for every call into that function. Imagine, for example, a program that called memcpy in 10,000 different locations - for callee cleanup there is only one instance of the stack management code for the memcpy function, whereas for caller cleanup there are now 10,000 copies. Even if this is only 10 or 20 bytes of code each time that's already hundreds of kilobytes of redundant code.

Another issue is stack growth. You mentioned that the compiler knows how much stack a function will use - this is not strictly true. Dynamic stack allocations are quite common, and your proposed model sandwiches the callee stack between the stack frame of the calling function and the return address of the callee:

| return addr | saved *bp | stackalloc space | return addr | saved bp | ...
|  (callee)   | (callee)  |     (callee)     |  (caller)   | (caller) |
 *bp+0         *bp+N       *bp+2N             *bp+?N         ...

This means that if the function requires its stack space to grow dynamically, it must move the return pointer and stack frame pointer (saved *bp) when it does so, and also modify the stack frame pointer in the process. This is unnecessarily complex and introduces some performance issues.

You may have also just realised why this isn't a fix for buffer overflows. By overflowing a buffer in the stackalloc space you don't overwrite the callee's return address, but you can overwrite the caller's return address. This does not really complicate exploitation.