I've been researching into some methods to ensure that disk encryption keys can't be stolen, and one that I've come across is called TRESOR. With this kernel patch, you can create a partition that is encrypted using AES, and the keys are stored in the CPU caches instead of in RAM. At first, I thought this was great, but upon further questioning, I've come to the conclusion that any decrypted data still stored in RAM is vulnerable to a physical cold boot dump. A potential way for this to be secure is encrypting the contents of usable OS RAM using TRESOR and then creating an encrypted partition, since the keys will be in the encrypted RAM, and the RAM's keys in the CPU cache. I still don't know if this is a secure method, or how sound it is. Is it even possible to prevent a physical attack on full disk encryption, and if so, how can I?
For the CPU to actually process the data, the data must be in cleartext at some point. TRESOR uses the CPU registers, presumably harder to read than RAM for physical attacker, but its scope is limited to the encryption key: that key is in the registers, but the data that is encrypted or decrypted is still in RAM. There are only a handful of registers in the CPU, there is simply no room for more data.
Using a TRESOR-backed RAM disk, you have "encrypted files" with fast access (since that's RAM), but they are still "files" in the view of the kernel and application code. The data will be decrypted in RAM at some point.
If you really want all RAM to be encrypted at all times, then the choice is either to use a CPU which includes the hardware to do such encryption (there are some smartcard CPU which do that, but no PC-like CPU to my knowledge), or to make a complete virtual machine whose virtual CPU is implemented TRESOR-like with automatic encryption and decryption upon each memory access. The actual encryption mode would need some thorough thinking (it is not easy to encrypt data securely while still allowing fast decryption with random access). I am not aware of any prototype; but it is theoretically feasible. A nice project for a student (it could even be the main result of a PhD). It would be awfully slow.
Well, there is some old saying from Microsoft (I know, not the best security source) that says:
And why is that applicable to your case? Because if you have sufficient cryptography, your hard disk is secure if someone steals it. But if the person can have access to your RAM chips right after your server was powered down, he can try to recover the keys in the scenario you just said.
Storing the keys in the CPU cache would minimize it, as you said, but unencrypted data could still be available in RAM chips.
And what if he has access to your computer before you power it on? Are you considering physical protection before booting up? otherwise, the bad guy can create some fake login screen and grab your password even if your whole system is encrypted. Or he can attach some keylogger. Or he can attach some device and clone your RAM chips from the beggining. Or he can change your BIOS and do that too. Or... well, take a look at the NSA catalog, for example, to have some ideas of the potential attack.
So your computer has to be physically secure. And by that, it means that not only physical access must be hard (for example, given enough time to the RAM content vanish), but also that he can't capture magnetic radiation from the CPU or motherboard or keyboard, nor noises, nor pictures from you typing or moving the mouse.
If you really need that level of security that taking out RAM chips out of your computer is necessary, you probably need even more security than you expect.
After your comment:
I think that I can expand a little my answer, just thinking in this physical aspect.
Encryption is your friend. If someone takes out a HDD that's encrypted with strong algorithm and strong password, he'll take forever doing brute force to attack it.
Points to consider:
a) whole disk encryption requires that you or some program "type" the password when mounting it. If it's you, you'll have to be available to turn it on every time. If it's a program, where to store the key
b) encrypt just data/folders, the same problem "where to store the key to turn it on" exists.
While running, you would need that the path between HDD - RAM - CPU is encrypted, as the answer from Thomas Pornin explains, and that I'm not aware of any server with a CPU able to do so. And you would need to worry about compatibility with some periphericals, since PCI, PCI express, firewire, etc., rely on DMA to speed up things. So, you could have performance issues.
An easier way to avoid that is, well, physically protecting your server. Like some safe room, where the server is powered down and warmed to certain temperature (to avoid someone stealing your RAM chips). Or even destroying chips with controlled blast or voltage peaks. Anything that make it harder for someone breaking in and go away with your server under his arms.
And, of course, all this leaves out the that many attacks can be done remotely, without any physical presence, just using software and so on.
I'm assuming that OP is building a server, and shipping it to the colo facility.
Physical security would include:
Software security would include:
That should do it, except against very determined and skilled adversaries.