I'm creating a messaging system where users send encrypted messages encrypted by the receivers public key. The receiver would decrypt the message using his/her private key. As the messaging service, is there any way that I can validate that the encrypted message has been properly encrypted by the receivers public key? Assuming I do not have access to the original plaintext message, is there any validation that can be done on the secret message to return an error to the sender if the message will not be able to be decrypted by the receiver's private key? The public keys will be accessible by my system.


No, it cannot

There is no such mechanism in RSA.

Even if you have the private key you can't check if a message was encrypted using the corresponding public key unless you know something about the plaintext (for example, that is should be readable ascii and not random bytes).

If you implement message authentication (i.e., the message is also signed using the senders private key), you can use the senders public key to verify that the decryption with the receivers private key was successful.

  • 1
    The second part of your answer relates to authentication and not integrity. A valid point , but not needed solely for integrity in which the question is looking for. – ISMSDEV Aug 18 '19 at 4:25
  • @ISMSDEV Yep, you're completely right. I would've added it to the question but there's already another (way better sourced) answer available – Jenessa Aug 18 '19 at 9:53

Integrity is what you are looking for. With integrity, the receiver can check that the message is valid. One can use Message Authentication Codes like EAX and GCM. Take care of the order of operations if you insist on this. Also, take care of proper padding if you are using RSA. For encryption PKCS#1.5 and RSA-OAEP and for signature RSA-PSS.

Normally, we don't send messages with the public key cryptography. We use hybrid cryptosystem in which the public key cryptography is used for key exchange as in Diffie-Hellman Key Exchange, DHKE, in fact, the Elliptic Curve version is better, ECDHKE.

Once a key is exchanged, the parties can send messages with AES-GCM. The GCM mode is an Authenticated Encryption (AE) mode that gives you confidentiality, integrity, and authentication. You can also use chacha20-poly1305 AE.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.