I'm creating a messaging system where users send encrypted messages encrypted by the receivers public key. The receiver would decrypt the message using his/her private key. As the messaging service, is there any way that I can validate that the encrypted message has been properly encrypted by the receivers public key? Assuming I do not have access to the original plaintext message, is there any validation that can be done on the secret message to return an error to the sender if the message will not be able to be decrypted by the receiver's private key? The public keys will be accessible by my system.
2 Answers
No, it cannot
There is no such mechanism in RSA.
Even if you have the private key you can't check if a message was encrypted using the corresponding public key unless you know something about the plaintext (for example, that is should be readable ascii and not random bytes).
If you implement message authentication (i.e., the message is also signed using the senders private key), you can use the senders public key to verify that the decryption with the receivers private key was successful.
-
1The second part of your answer relates to authentication and not integrity. A valid point , but not needed solely for integrity in which the question is looking for.– ISMSDEVCommented Aug 18, 2019 at 4:25
-
@ISMSDEV Yep, you're completely right. I would've added it to the question but there's already another (way better sourced) answer available– JenessaCommented Aug 18, 2019 at 9:53
Integrity is what you are looking for. With integrity, the receiver can check that the message is valid. One can use Message Authentication Codes like EAX and GCM. Take care of the order of operations if you insist on this. Also, take care of proper padding if you are using RSA. For encryption PKCS#1.5 and RSA-OAEP and for signature RSA-PSS.
Normally, we don't send messages with the public key cryptography. We use hybrid cryptosystem in which the public key cryptography is used for key exchange as in Diffie-Hellman Key Exchange, DHKE, in fact, the Elliptic Curve version is better, ECDHKE.
Once a key is exchanged, the parties can send messages with AES-GCM. The GCM mode is an Authenticated Encryption (AE) mode that gives you confidentiality, integrity, and authentication. You can also use chacha20-poly1305 AE.
