I am looking for a way to encrypt certain files (.jpg, .pdf, standard file extensions) such that they can only be opened by a certain application (assuming this application will have the relevant file viewer included with it).

It would seem that the way to go about doing this is to have some encryption scheme specified by the application client; to encrypt a file, it must be done through the client - thus to open the same file, only the application can understand how to open it.

How should I go about trying to implement this application-specific scheme, using standard encryption techniques (if possible)?

Pardon my ignorance, I am very new to information security.

  • 1
    Another application running under the same user account could access your application's decryption keys (from disk, or depending on the system it could attach itself as a debugger and read them or the decrypted data from the process memory). If you need applications to be able to keep things private from each other, then they need to be run as different users or in sandboxes.
    – Macil
    Jul 14 '16 at 21:13
  • 1
    On Linux, AppArmor or SELinux might allow you to keep other applications from reading your application's encryption keys, but at that point you might as well forget about doing any encryption and just use AppArmor/SELinux to protect the files themselves.
    – Macil
    Jul 14 '16 at 21:15

There are several ways to do this but basically you are describing something of a digital rights management problem.

There are some open-source DRM programs like OpenIPMP which may help but there are probably other ways to solve your problem without using DRM. https://sourceforge.net/projects/openipmp/

You didn't provide a lot of information about your needs or limitations but if it's an option the application could simply send the data to a remote site for decryption such that the application itself would never have access to a copy of the decryption key at any time. This could even be implemented as something of a streaming service of sorts.

Similarly you could implement a type of certificate-based PKI USB authentication token (hardware key). Again this type of solution is aimed at protecting the decryption key from other applications.

There are probably hundreds of ways to solve this problem but you may have to look at your limitations and options from a different perspective.

As much as I'm not a fan of DRM software solutions recently I've been seeing an increased need for easy to implement / free / open-source DRM solutions. It's an interesting problem to have.

  • How exactly would 'sending to a remote site' work in practice? The user's client, when the user is attempting to open the file through the application, would send a decryption request to the server, which would respond with the corresponding key?
    – socrates
    Jul 15 '16 at 18:11
  • Send encrypted data to the remote system where it is decrypted then returned in clear-text just to the browser/application for temporary use by your application but not cached/saved to disk. The important part is you could build controls to prevent other applications/people from accessing the decryption key this way. Jul 15 '16 at 18:13
  • that make sense, but the encrypted data would be files, which may need to be saved to the disk, and may be large; the temporary use makes sense for certain cases i can think of, but not others. but i guess this mechanism (client request -> server approved -> server decryption key -> client -> unlock file) still ensures safety, if the client then just uses the returned key to decrypt the file locally?
    – socrates
    Jul 15 '16 at 18:18
  • Yes, you could technically make it two-way (read and write) to add encryption if you want too. You'll need encryption in transit of course but it's not that different than building a web-based service in ways. Again this is just another approach option. Jul 15 '16 at 18:24

Yes, but don't do it.

If you are asking if it is possible to come up with a proprietary scheme for encrypting data that only your application "knows," the answer is yes, until someone figures out how to reverse-engineer it (which is usually easier than you realize). This sort of practice is known as security by obscurity and is highly discouraged.

Instead, your application should use an encryption scheme that is public domain (and therefore proven and well-tested). To provide application-specifity, use a cryptographic key that only the application "knows." A hacker attempting to reverse engineer the key may have some success, but the paramters governing that success (e.g. how long it will take and the computing power required) are well known and understood and you can devise an overall risk strategy appropriate to the level of security required by your business case (e.g. you can use longer keys or change them more often).

Encryption is only part of the problem

Not sure what attack vectors you are attempting to mitigate, but I'm guessing you want an "image viewer" application that is the only means to viewing an image, and in addition you are working with the assumption that the application can fall into a hacker's hands. If these are correct, you are up against a very difficult problem (compromised client endpoint).

Usually if there is a compromised client endpoint, us security professionals give up. A fully bullet-proof solution is nearly impossible. But attempts have been made, and these fall under the lexicon of Digital Rights Management (DRM). Here's a few of the problems you will need to solve:

  1. You need a protected media path.

Even if all the security in your app works, it still needs to display the image, and the way your O/S displays an image is using a video driver. Well, anyone can write their own video driver, and there is nothing stopping a video driver from recording and storing an image for unintended use. The typical way to deal with this is to require the video driver libraries to be digitally signed, and for the O/S to verify the signature before allowing output to go to the driver.

  1. You need to protect your application's memory

No matter how clever your app is, a hacker can find a way to get access to its memory if he has physical access to the machine. A bare minimum mitigation for this is to run a little code to check to see if any debuggers are currently running on the system, and halt output if there are. This is not likely to stop a determined hacker who has ways to mask such things.

  1. You need to protect the content and the key

Solutions I have seen involve storing the media in encrypted format (using a symmetric key-- a public/private key is way too slow). When the viewer runs, it requests a key from a server, using a signed document indicating that it has a right to request the key. The key is stored in memory and the content is decrypted, then the key is overwritten in memory with random data and discarded.

  1. You need to stop the hacker from taking a photograph of the video monitor

Pretty hard problem to solve. Even if you do all this security stuff, nothing in the world will stop someone from pulling out their camera and taking a shot of the screen. This would be lossy, of course, but with a good monitor and good camera you can still get a great image.

The only mitigation for this I can think of is to include a contrast-sensitive watermark on the image itself. This may or may not be acceptable to you.

  • ah, okay; so we basically treat the application itself as an entity with its own private key, then encrypt all uploaded files with its public key, correct? How does one keep the private key private? it would presumably have to be built in somewhere into the compiled application, correct? couldn't theoretically a hexdump reveal the plain text of the key? (theoretically, as this is highly impractical)?
    – socrates
    Jul 15 '16 at 18:08
  • Any overall solution is going to be very complex; I've edited my answer to give you an idea what you're up against.
    – John Wu
    Jul 15 '16 at 21:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.