Is there a way to secure a byte[], PrivateKey, PublicKey, KeyPair, and SecretKey in an application so that a malware would be unable to retrieve the key by analysing the memory? Or at least, how to harden it so that it is more difficult to retrieve the ephemeral keys (no "protection" using hard-coded keys please)?

Or is it pretty much a "goner" if the attacker has access to the device?

Language is Java, so please assume that the attacker has full knowledge of the source code.

Without using HSMs and other similar devices.

  • there are some tricks to protect data in RAM, but they are very expensive (in terms of implementation) and are definitely beyond the standard methodology which is to mark the bytes in RAM as "please don't swap" for the OS. – SEJPM Mar 22 '16 at 14:56
  • But generally speaking: You can't protect against your host (without specialized hardware which then becomes your host). – SEJPM Mar 22 '16 at 15:40
  • Is there a way to do that in Java? – user3635998 Mar 22 '16 at 21:48

You can't protect against your host, be it the OS, the processor or the BIOS.
Also see the Laws #1 and #2 of Security.

This doesn't mean though, that there's no way to defend yourself against at least some software attacks. Please note though: If you really want to protect the key, you have to use hardware-enforced technologies like HSMs, TPMs, smart cards and SGX. The first two methods I'll outline are basic and are done by every somewhat decent cryptographic library, the latter ones are more exotic.

Defense #1: Always clean up behind yourself!

It is generally considered best practice to always clean up whatever mess you create in programs. This also applies to security engineering and thereby you really should reliably zero out every key when you don't need it anylonger. Note that a simple C-style memset(pointer,0,size); won't do the trick because the compiler most certainly will optimize this away, you should use dedicated functions for burning your keys in memory (Windows offers them and you can write them yourself - in assembler or as a simple loop).

Defense #2: Never stop watching your keys!

Modern operating systems commonly ship with a feature called "swap-memory", which is great for the application developers as this can increase the size of the RAM when needed by swapping out pages to the hard drive. This however is really bad for security, because wiping the keys after they have been swapped out tends to be rather difficult and thereby you should tell your OS through an appropriate function call to always keep your keys in (physical and ephemeral) RAM.

The following defenses are more advanced and it is less likely they are deployed anywhere. They are however listed in Schneier's Cryptography Engineering and can defend you in some circumstances. Let the tinfoil hattery begin!

Defense #3: Encrypt all the things!

You may have heard about "cold-boot attacks". Basically they are the (unlikely) scenario, that an attacker can get hands on your machine, cool it down (more or less) and extract the secret cryptographic keys for several seconds or even hours. It is very hard to defend against such attacks (which also include (automatic) memory dumps). This is why the Boojum (this is the actual name!) was invented.

Say, you have secret keys k,l,m,n, each of length 128-bit. Now you create another secret key i, which you use to AES-encrypt k,l,m,n in CTR or GCM mode (authentication doesn't matter too much here). Now whenever you need k,l,m,n, you just AES-decrypt it and use it and re-encrypt it or wipe the temporary copy. The question now becomes "How to protect i?" With the Boojum, you create a random string R in RAM at a physically distant memory location than i. You hash R and XOR it with i and store that. Whenever you need i, you hash R again and XOR it with the stored value to recover it. Of course this isn't too secure if you keep R the same all the time, because the RAM will eventually "learn" the value and keep it longer, so you need to update this often. You update it, by (optionally) selecting a new memory location and generating a new R' and storing R' XOR R at the (new) location in memory. Now you unmask i using R's hash and hash the "new" R and mask i again. Of course you delete the old R. It is recommended to repeat this update process frequently, e.g. every second.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.