Protecting passphrases within an application

Question

First off, I'm not very familiar with encryption best-practices.

My use-case is simple: I'm creating an application that will be reading encrypted documents and decrypting them dynamically to do some operation on the data that they contain.

I was reading that I should use 128-bit or 256-bit AES, so I installed OpenSLL to my Linux box to see how encryption / decryption is done. I encrypted and decrypted a sample document and notice that AES requires a passphrase.

How can I store this passphrase in my application securely so that somebody cannot easily reverse engineer the binary, discover my passphrase, and decrypt the documents outside of the application?

Answer 1

What you're talking about, essentially, is DRM. If you provide the user with the data you're trying to protect in any unencrypted form, they have the data and can make copies. It's on their machine, you've given it to them.

• If you use AES, or any form of symmetric cryptosystem an attacker can reverse-engineer your application and discover the key.
• If the key is only provided at runtime, they can load your program into a debugger and set a breakpoint to capture the key.
• When you decrypt the document, anyone can set a breakpoint on your decryption routine and capture the plaintext buffer from memory.

Packing, obfuscation and anti-debug tricks are trivial to bypass for anyone that's even reasonably skilled at reverse engineering. This is self evident; think about every major game / software title that's come out in the last 20 years. Every copy-protection mechanism has fallen to reverse engineering. The same goes for malware and media DRM.

All of this boils down to one thing: Anything you put on your client's computer is theirs to modify and analyse.

I have a feeling that your situation needs to be re-thought. Rather than coming up with a potential solution and trying to find ways to make that work, think about alternative solutions. Can the document processing be done on a central server, out of the hands of the user?

Answer 2

Generally you can't stop a determined or experienced hacker. My suggestion is not to worry too much because unless you have a lot of users (or money) it is unlikely anyone is going to put any serious effort towards hacking your application; Concentrate on delivering a fantastic piece of software.

However you can:

Avoid storing passwords as string literals - This should be obvious.

Obscure your passphrases - Use a collection of plain english words or random characters to buffer your passphrase to possibly confuse the hacker or to make the passwords less obvious.

Obscure symbol names - There is a possibility a hacker could extract obvious symbol names from your binary. For example, class keyChain. You can mislead while maintaining readability by using #define keyChain readModule.

Obscure application flow - Say the hacker works backwards from the file decryption to find the passphrase, you could use function pointers to obscure the appeared application flow.

Generate your passphrase at runtime - You write your own function that generates and spits out your passphrase during runtime.

Encrypt / Decrypt passphrase with your own function - If you use another third party library an attacker can look for common calls. Obviously be smart with this and don't use some form of the caesar cipher, and remember to only use this to hide your passphrase, that is, still use AES from OpenSSL for your actual data encryption and decryption. However this will lead you back to the start of safely storing a password. The gain is in the attempt to prevent the hacker from searching common decryption/encryption function calls.

Regardless, none of these will completely stop a hacker. They may slow down or prevent a novice, or they may do absolutely nothing.

Work on making your software that great that bypassing it straight to decryption is unattractive. Convince users that your software is worth using.