I've been contemplating the fundamental issue of stack overflows, from the classic shell code injection that doesn't work with NX bit to the newer ROP gadgetry.
The first question I had was in regards to CPUs that have branch registers. It seems as though doing any of these sorts of attacks on a CPU that tracks branches in a "stack" of branch registers, like Itanium does for example, would be difficult if at all possible. Obviously there are other attacks that don't require overwriting the return IP but doesn't the combination of branch registers and an NX bit foreclose a huge swath of attacks? If that's the case, why don't people consider using processor architectures that employ branch registers for their edge devices?
As a follow on question, why doesn't Intel/AMD introduce a mode of operation in which the series of CALL/RET instructions are verified and enforced? It seems the real problem is that the CPU leaves the tracking of return IPs in the care of the process, on the stack. It would be a very bad compatibility break to simply not push the return IP onto the stack, or even to push some random value, but why not offer a mode in which the CPU on CALL pushes the return IP onto the stack AND onto a CPU managed backing store for this process/thread/context. These pages would be totally off limits to the process except through CALL and RET. When RET is executed, the address popped off the stack is compared to the address in the secondary storage and if they're not equal then abort. Obviously this would be slower, but for security sensitive parts of applications, such as those which handle external input, the application could set the safe flag/mode on choosing to sacrifice performance for security.
Does this concept make sense? Would it actually mitigate return IP oriented attacks?