Is encrypting data with a private key dangerous?

Question

I have been reviewing code for various internal apps that handle secure transmission of data with external systems. These follow the following process:

• RSA encrypt data with 3rd party public key
• RSA encrypt again using our own private key
• Transmit data
• 3rd party decrypts using our public key
• 3rd party decrypts again using their own private key

In this system we generate new private/public keys for each third party and all public keys are distributed securely and then kept private.

I don't understand why we would encrypt data using our own private key. I suspect the designer's intent was to enable verification that the data originates from us and that same person had no knowledge of signing messages to implement that instead. Unfortunately, that person is no longer around to be asked.

I am happy that our data is acceptably safe because we encrypt that with the third party key, but private key encryption seems sufficiently non-standard that I feel some discomfort. Is there anything that I should be concerned about?

Answer 1

It seems that the code you are looking at is taking a bit too literally the first description of how RSA signatures work.

When RSA was first described, it was an asymmetric encryption system: sender encrypts data with the recipient's public key, and the core operation is a modular exponentiation which takes as input a number modulo n (n is the modulus, a part of the public key) and returns a number modulo n. Then somebody said: hey, if we reverse the process and decrypt some input data (with a private key, necessarily), then we get a signature since anybody can perform the reverse operation (encrypt with the public key), which counts as signature verification.

This is an over-simplistic description of RSA signatures, and it does not actually work, because it neglects the effect of padding, which is necessary for security (see PKCS#1 for the Right Way to do RSA encryption and signatures). But the idea that signatures are encryption-in-reverse (or "encrypt with the private key") is unfortunately widespread, and lead to scores of homemade solutions which are all, invariably, weak in some way (but not necessarily in an obvious way). Chances are that you observe such a homemade solution.

Answer 2

It's at least bad practice, and probably dangerous. I strongly recommend changing the scheme you use.

You shouldn't encrypt the content with your private key. You should sign it with your private key. While signing and encryption with private key are similar operations when using RSA, they should be considered separate. In particular signing works on a hash, not the actual content.

Depending on how exactly your current system is implemented, it might be possible for one third party to trick you into decrypting data received from another third party.

It might also be possible to trick you into signing something you didn't want to sign that way. But without the details of the scheme you use, I can't say for sure.

Answer 3

PKI algorithms generally use longer keys and thus are more computationally expensive. That's one reason things like SSL/TLS use PKI to setup the stream but then use a shared key cipher with a shorter key.

Whether it's risky to encrypt someone else's text with your key depends on the cipher being used. Some ciphers are vulnerable to chosen-plaintext attacks. Plain RSA is one of these. PKCS#1 isn't.

But as you say, if all you need is authentication, you can just sign it (thought that requires your private key as well.)