The problem with most operating systems is that they follow a specific "calling convention." This convention requires putting function parameters on the stack, being some derivative of the C-style calling convention. You must use this convention for ABI (Application Binary Interface) compatibility with that OS. So, without OS support, you could only use this feature for calls made within your application.
This would complicate compilers quite a bit and probably require a fair amount of work. In short, you could protect your own programs if you had a compiler that supported this calling convention, but you'd still be at the mercy of the OS whenever you had to do things like reading/writing a file, etc. A buffer overrun in a DLL, for example, can't be fixed by you changing your calling convention.
Secondly, until recently, with the advent of virtualization, it really wasn't feasible to set up a separate area like this, because segmentation was expensive, and memory virtualization even more so. Today, this would practically be a non-issue, but since we have to deal with historical software (e.g. stuff written ten years ago that still require the conventional calling methods), the OS would then be forced to support both models for some indefinite period of time.
If a new OS with no compatibility concerns were written, it could certainly do this, but it probably won't happen, because there are more viable methods. Microsoft's own Singularity OS is completely immune to buffer overruns (according to them), because the OS statically validates that each program cannot possibly misbehave. Interestingly, this OS uses no "memory protection" as used by Windows, Linux, Mac OS, etc. The programs are validated for correct behavior before they run, not as they run. Of course, if a virus were capable for this system, it would have unlimited system control because of the lack of protection at the hardware level.
In short, even without any serious research on the topic, it's clearly possible that this approach would work, but outside of FOSS (Free and Open Source Software), it wouldn't be possible for this approach to work from a financial standpoint. Linux could be re-written from the kernel up to support the new model, a compiler rolled out for it, and then all the software out there could be re-compiled without too much effort (note: "too much" being relative to, say, Microsoft).
Microsoft, Apple, and so on do not have this benefit, because the code is already compiled by millions of different developers, so anything that couldn't be updated would be instantly obsolete, or they'd have to write emulators to support the old software. Basically, until someone comes up with an OS that has this feature built-in, with compatible compilers (at minimum C and C++, plus probably Cocoa and Win32 C++), and it gains enough support to become a contender against Linux, Microsoft, and Mac OS, it'd be pretty hard to justify moving to a new model. Linux would be the easiest to move over, although all software would have to be compiled until RPMs and other package types supported the new calling model.
Finally, DEP (Data Execution Prevention) pretty much solves the problem in most cases, making it harder to execute code that shouldn't. This also reduces the need to switch to a new model, although, as Singularity demonstrates, hardware could be a lot faster if it wasn't constantly forced to protect against programmers' bugs and the exploits they present.