I've been contemplating the fundamental issue of stack overflows, from the classic shell code injection that doesn't work with NX bit to the newer ROP gadgetry.

The first question I had was in regards to CPUs that have branch registers. It seems as though doing any of these sorts of attacks on a CPU that tracks branches in a "stack" of branch registers, like Itanium does for example, would be difficult if at all possible. Obviously there are other attacks that don't require overwriting the return IP but doesn't the combination of branch registers and an NX bit foreclose a huge swath of attacks? If that's the case, why don't people consider using processor architectures that employ branch registers for their edge devices?

As a follow on question, why doesn't Intel/AMD introduce a mode of operation in which the series of CALL/RET instructions are verified and enforced? It seems the real problem is that the CPU leaves the tracking of return IPs in the care of the process, on the stack. It would be a very bad compatibility break to simply not push the return IP onto the stack, or even to push some random value, but why not offer a mode in which the CPU on CALL pushes the return IP onto the stack AND onto a CPU managed backing store for this process/thread/context. These pages would be totally off limits to the process except through CALL and RET. When RET is executed, the address popped off the stack is compared to the address in the secondary storage and if they're not equal then abort. Obviously this would be slower, but for security sensitive parts of applications, such as those which handle external input, the application could set the safe flag/mode on choosing to sacrifice performance for security.

Does this concept make sense? Would it actually mitigate return IP oriented attacks?


Your idea sounds like ROP Defender although that is a software implementation.

This would make buffer overruns harder, but while I can't think of a way round it just now, I expect some smart exploits could get round it. So what you propose is effectively another kind of anti-exploitation technique - it would be a sister to Stack guard, DEP, ASLR, etc.

When considering anti-exploitation techniques there are two main considerations: compatibility and performance. These techniques are only used when there is a very good pay-off. My gut feel is that this won't cut it - too much performance pain for too little benefit. But who knows, maybe it will get implemented.

| improve this answer | |
  • If you were only paying the penalty in input parsing and other security sensitive areas, perhaps it would be acceptable. – Jeremiah Gowdy Nov 29 '13 at 22:37
  • @JeremiahGowdy It's quite hard to isolate security sensitive areas as untrusted data can spread far within a program. However, Stack guard has some heuristics for when it doesn't need to protect a particular function (largely based on what local variables there are) - so I expect you could do the same. – paj28 Nov 30 '13 at 18:23

How do you expect the CPU to verify these matches? It would have to memorize the location of call instructions. Where would it put them? on a stack? Processors don't have variable-sized storage.

I don't know where you get that a CPU “tracks branches in a ‘stack’ of branch registers”. This is not the case at all. CPUs maintain branch prediction data, but they don't track in what order the branches were taken.

Even if the CPU somehow found a place to store branch address, it would have to be told about all the things that break the call/ret matching: nonlocal returns (exceptions), context switches, etc.

In addition, many (most?) hardware architectures don't even have call/ret instructions! At least on ARM and MIPS, a function call is a push followed by a branch, and a function return is a branch followed by a pop. (I'm simplifying but you get the idea).

I don't know about the Itanium feature you mention, but if it can really do what you're after, it's highly unusual.

Simply put, what you're proposing cannot work with a typical hardware processor. It might work in some virtual machines.

| improve this answer | |
  • Itaniums have 8 branch registers, and when a CALL happens, the return address is pushed into br0 rather than being part of the stack frame. Itaniums also have "stacks" of registers which can "spill over" into backing store (memory) when needed. Here's the good description of the IA-64 architecture I'm talking about. msdn.microsoft.com/en-us/magazine/cc301711.aspx – Jeremiah Gowdy Nov 29 '13 at 22:28
  • @JeremiahGowdy Thanks. So what do you do at the 9th nested call? The stack spills over to RAM. So at best an exploit that requires overwriting the stack doesn't overwrite the return address for the current function, but an address a little higher up the chain. That doesn't make the exploit significantly harder. – Gilles 'SO- stop being evil' Nov 29 '13 at 22:33
  • Also Intel x86/x64 processors already support a similar feature, Last Branch Records which I believe lets you track up to 16 branches. As far as the variable storage, the CPU is already operating the stack for you in CALL/RET with appropriately configured stack registers. There's no reason you couldn't flag another page or two as your call stack mirror area. – Jeremiah Gowdy Nov 29 '13 at 22:33
  • There is plenty of other memory that can be pointed at besides "the" stack. That's my point. The colocation of local variables and the tracking of the call stack leads to this category of exploits. – Jeremiah Gowdy Nov 29 '13 at 22:34
  • @JeremiahGowdy Whatever other memory you'll be pointing to will then be the stack. Still doesn't make any difference. You could avoid some classes of exploits by going to a Harvard architecture, with pointers to code and pointers to data living in separate memory. You can't do object-oriented programming any more, though, and ROP remains possible. – Gilles 'SO- stop being evil' Nov 29 '13 at 22:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.