I also would like to know if it would be theoretically possible (whilst obviously difficult) to translate an existing JIT compiler (e.g. .NET CLR) that produces code that automatically encrypts its user-mode memory.
This might be quite difficult for the OS to do on behalf of a program. The reason for this is that the virtual pages allocated by the operating system are in fact controlled by the CPU - the OS instructs the CPU as to what it wants in terms of storage areas and the CPU then computes the offsets on that basis for each program and applies the result directly. Have a read up on the MMU and Paging on x86.
In short, when your program makes the computation
lea eax, [edi+something], the MMU/paging handles the address translation and issues a page fault when the page is not found. That allows you to then load the page out of storage, should you need to.
So the memory access doesn't go through the kernel per se and as such can't be hooked and processed as a file read or write can (your read and writes are translated through the appropriate system calls into the appropriate file system structure. As such, the OS sees the data as it passes through. You don't need a system call to write to RAM. You could make one, but it would only work for programs that call it and so not most programs).
However, that doesn't mean to say a JIT process couldn't do this on behalf of an application, or the application itself couldn't keep data encrypted until it needs to load it into registers, decrypting it as it passes over the data.
In that case, you're into the issue SteveS covers well - you have a key storage problem. The key itself must also be in RAM somewhere. This is a turtles all the way down problem - fundamentally it's impossible to keep that key "secure".
Finally, since being able to read another application's RAM requires supervisor mode access to the CPU (i.e. kernel space) you probably have bigger issues if your concern is software interception. If your concern is hardware, I think physical security might be a better way of mitigating the risk.
Edit I had a look for papers. Here's one called CryptKeeper. Their technique is to have a large encrypted "RAM disk" as the swap file and swap all their pages out to that when not used:
We mitigate this vulnerability with Cryptkeeper (CK), a
software-encrypted virtual memory manager. Traditional pro-
cessors cannot operate on encrypted data, so CK segments
RAM into a smaller working set called the Clear, and a
larger encrypted RAM device called the Crypt. As the working
space fills, pages are automatically swapped into the encrypted
portion of memory, and are decrypted on demand.
Apparently the performance isn't too bad, but I don't think there are any implementations of it yet.
Edit 2 So it turns out CryptKeeper and the OpenBSD Swap Encryption mechanisms work on a very similar level; it does not actually encrypt physical memory but uses physical memory as a swap structure, forcing page faults and encrypting/decrypting data in resolution.