I want to create a message exchange program. These messages are encrypted with the AES key, and this key is encrypted using the RSA algorithm. When I send a message from User 1 to User 2, I need to request the public key from User 2, do I just get the key and use it or do I need a secure key exchange like Deffie Hellman or something else? If the answer is yes, how will I be able to use the deffie-Hellman algorithm to send this RSA public key? What I read that Hellman's algorithm generates its own public and private key, and both are dependent on it, then how do I pass the RSA public key?

The second question is whether I encrypted using the AES algorithm only, and here I have to send the public key from the sender to the receiver, then of course I need a secure medium to transfer the keys like Deffie Hellman's algorithm. but as I mentioned before that this algorithm generates its own public and private key. But I already have the AES public key. Is there a way to pass the AES public key to deffie hellman? please help me I am really confused about that thnx

  • Have you looked at Noise? I've been trying to do something similar lately, and I got a lot of useful knowledge after reading the spec a few times. The Noise homepage also links to some libraries that you can use instead of implementing it all yourself. – 5ndG Aug 19 at 16:01
  • It's not really clear what you're trying to do. What's the relationship between you, user1 and user2? Who's doing the decryption? Are you doing end-to-end encryption? – Steve Sether Aug 19 at 21:25
  • I think you need to answer some of the basic Ws. Who, What, Where, Why? You want the How, but that answer depends on all the other Ws, which we don't know. – Steve Sether Aug 19 at 21:37

TL;DR: Skip to "Using result of DH".

Because you started with "I want to create a message exchange program" together with "I don’t know how to use DH", a tedious preface:


People trust secrets to secure information systems and when those systems turn out to be insecure people get hurt, sometimes die. We want secure systems that are actually hard to break, even for a competent adversary.

What if your system is not meant to protect anything from a competent adversary? Then why are you building a system that is a messenger with RSA and AES? Obviously to advertise it as secure. Don’t advertise something as secure when it is not secure, because people will trust secrets to it and then they will get hurt.

You think your system is secure? It is not. The definition of "secure" is "multiple people who have a proven good track record of breaking comparable systems have tried to break this system for a long time and got nowhere". Because attacks only get better, even issues that do not immediately lead to a full break make the system broken or at least not recommended.

You think that standard is too high? You want me to break your system right in front of you and unless I can do that, you’ll call it secure and me jealous? The people who are capable of breaking non-trivial systems (I’m not one of them) don’t work for free. You might get a sketch of a way to break it for free, but without a lot of studying you won’t even understand what they mean. Professional security audits by real cryptographers and security engineers are expensive, you are in a bidding war for their time against financial institutions and the like.

A way to see your system get broken for free is to make it high value. Using the system to coordinate terrorist attacks will get the NSA on it, but they won’t tell you how they broke it. A system that would let an attacker steal a few bitcoins might attract qualified attackers.

The traditional way to get cryptographers work on breaking your system for free is to get an advanced degree in cryptography (or equivalent), publish breaks (or at least publishable results) in journals and conferences (they accept good papers by unknowns), become a known cryptographer, then publish your own design for others to break. Competitions like NIST LWC are a good venue.

What for?

Why do you want to build a secure messenger app? There are plenty of existing apps to choose from. Signal is the gold standard of secure messenger apps, it goes above and beyond to not let the server know anything about you. Even its controversial use of phone numbers is because they don’t want to store your contacts list on the server.

You mentioned RSA and AES.

Because they are household names. Non-experts just trust brands they recognize, regardless of all the information an expert would use to evaluate a product.


RSA is obsolete. It is not used in new designs. It has a number of problems:

  • It exposes a footgun in the form of "encrypt to public key" which invites people to build systems without PFS.

  • It needs many bits to be secure, this makes it somewhat inefficient. To get 256 bits of security, you need impractically many bits.

  • It is under-specified in the standards, so you won’t find a misuse resistant RSA API that makes all the security decisions for you (e.g. Public exponent? PKCS#1 v1.5? PSS salt length? MGF1 hash function?).

  • For those foolish enough to implement RSA on their own, there are a great many more issues, chief among them is that many people were shown textbook RSA as a throwaway nifty thing in math class, without all the details that make it secure, and countless people have just implemented RSA insecurely based only on that information. But professionals mess up too.

What has superseded RSA? First, NIST P-256 ECDH, ECDSA and ECIES early in the 21st century and then X25519 and Ed25519 in about 2012.


AES is secure, when implemented well in hardware or when you use a slow (but secure) bitslicing implementation. Fast implementations of AES in software usually leak the key via side channels. Intel chips have had good AES for 10 years, so problem solved. There is a catch though: "if you have to type the letters "A-E-S" into your source code, you’re doing it wrong".

AES is a block cipher, the only way to use it is in a "mode of operation" (there are so many to choose from!), those need fresh IVs (TLS did it wrong!) or nonces which you must not repeat (the TLS site of the GCHQ did it wrong!), and possibly padding (padding oracles are a classic and the go aws s3 encryption sdk did it wrong!), after that the encryption is malleable so you have to add message authentication (iMessage did it wrong!), then you need to combine the encryption and the authentication correctly (TLS did it wrong!), then you need to verify the authentication tag in constant time, etc. etc.

The way these problems have been solved by everyone is by using an AEAD. The most popular one is AES-GCM (because it is fastest among the non-patented), but it requires hardware support and is easy to mess up, so I recommend using ChaCha20-Poly1305 (or XChaCha, which is even better). Those two options are what TLS, SSH and IPSec converged on. WireGuard, the modern replacement for IPSec, offers ChaPoly only, for extra security by eliminating cipher suite negotiation.

What about DH and PFS?

In the old days, people thought it’s a good idea to use the identity key as an encryption key, usually a key-encryption-key in a hybrid encryption scheme. The way that works is that everyone has a single keypair that they intend to use for decades and not lose the private key and also not get it leaked (so should you backup the key or shouldn’t you?). You can use your own private key to digitally sign things so that others can verify your signatures offline, and you can use the other person’s public key to encrypt things so that they can decrypt it using their private key, offline. This is how PGP works and this is also how the TLS RSA cipher suites work.

The major problem with that way of doing things is that when the key inevitably leaks, the adversary can decrypt all previously recorded intercepted messages and also impersonate you to your friends and sometimes even impersonate other people to you. People impersonating you is an unavoidable risk of losing your key, but decrypting previous messages is catastrophic and new designs that don't provide PFS are non-starters.

With TLS, this means the key for a certificate is valuable even years after the certificate has expired and cannot be used for impersonation.

PGP/GPG is not recommended by infosec professionals, HTTP/2 has disallowed the non-PFS TLS cipher suites in 2015 and TLS 1.3 has finally removed the non-PFS cipher suites in 2018. AFAIK, SSH has never been used without PFS.

The way this is solved, and the way to get PFS, is by first using DH. Each side generates new single-use (called "ephemeral") keypairs which are not linked to any identity, and they exchange public keys in the clear. Then one computes "g^(ab) mod p" and the other computes "g^(ba) mod p" and those two values are the same and nobody who saw their exchange can know this number. This is basically a magic trick, it takes people a long time to believe asymmetric cryptography is possible.

Using result of DH

The result of performing the DH key exchange is not a smooth uniform array of bits of the particular length that you need to use an AEAD so you pass this value through a KDF (HKDF is fine) and extract as many bytes as you need. For example, the first 32 bytes are the client’s sending key and the next 32 bytes are the server’s sending key. The client is the side that initiated the network connection or sent the message. The server uses the client’s key to decrypt the client’s message and its own key to encrypt its reply.

Now, this is an anonymous secure channel. Nobody knows what’s inside, but the two sides don’t know who they are talking to. The keys they used are not linked to anything, and they might be Man-in-The-Middled and they have no way to know. The first thing they should do using this channel is to use asymmetric authentication to verify they are talking to someone who holds the long term identity key they expect (or the long term identity key is itself vouched by someone they trust, like in PKI). This can be done using digital signatures (the usual way) or using DH and HMAC, the way the Noise Protocol Framework does it.

After they have performed authentication, they can exchange regular messages.

Note that the PGP way of doing things does not require a message round trip or for the recipient to be online. Simple DH requires a round trip and requires both sides to be online at the same time. The round trip can be avoided by the recipient publishing a bunch of signed DH public keys on the server, which the sender can use to encrypt messages and store them on the server for the recipient to retrieve when they are back online. This is what Signal does (including whatsapp). The Noise Protocol Framework enumerates a bunch of "patterns" and the features you get in exchange for more round trips.

Classic DH, today called FFDHE, has some of the same problems as RSA, and has been superseded by NIST P-256 ECDH and later by X25519, but conceptually they work the same way.

How to build?

The people who have learned to not roll their own cryptographic primitives sometimes roll their own cryptographic protocols, and then in the joinery between otherwise secure primitives they create game-over bugs.

What should we do instead of rolling our own? Use good out-of-the-box solutions, the higher level the better. If you can just use WireGuard or correctly configured TLS or SSH (disable all the backwards compatibility and leave single secure choice), or age and minisign, you should do that.

If you have to use a library, use libsodium or Google Tink. Both try hard to have an API that is hard to use insecurely.

If you use those and don't use any cryptography outside of what's inside those, you have a much higher chance of not having a major cryptographic issue.

Secure messenger

Suppose we have solved the problem of exchanging messages. Secure messenger apps have more features than that.

If users have accounts they can recover after losing their device, password recovery (or equivalent) is a popular way to get game-over bugs.

Replacing a device, even if you have access to your previous device, or adding a device to be logged in on two devices simultaneously, is a way to lose custody of your private key if done incorrectly.

The question of how the key directory service works is critical, because without ability to verify the public key of the other party, you cannot trust the system. If the server can replace public keys then it can MiTM users.

Group chats are more complicated because you need to rekey when group membership changes.

If you want the server to store the user’s contact list, this is a privacy problem. Storing group membership on the server is a privacy problem. Storing user profiles on the server is a privacy problem.

Various messenger apps solve these problems in various ways.

You should study the Signal Protocol whitepapers to learn the state of the art.

Please do a lot more reading before you implement anything anyone might accidentally use for anything important.

openness to new people and ideas

What if you have a brilliant idea and I won’t find out because I was dismissive of your apparent lack of knowledge about the current state of the art? That’s a hard problem. People will not invest the time to learn about your brilliant new idea, unless you demonstrate first that you know enough about the existing ideas.

| improve this answer | |
  • "Also, group chats are more complicated because you need to rekey when group membership changes" What if you use something like Shamir Secret Sharing? – nick012000 Aug 19 at 4:31
  • 2
    @nick012000 then that is more complicated but in a different way. It's also not immediately clear to me how you turn it into a solution, although that may be lack of imagination on my part. Your goal with group chats is generally to ensure that the group of people who can understand and post messages in the chat, is exactly the same as the group of people who are currently in the chat, even if that changes, and that a subset of that group can add or remove people from the chat. That's fiddly, and even Signal had a bug in their initial implementation. – James_pic Aug 19 at 8:59
  • 1
    @nick012000 If the people in the group have the key, then the people who leave the group still know the key. Unless you change it. End of story. – user253751 Aug 19 at 10:22
  • 2
    This answer is technically very good, but the information is portrayed in a very snarky and negative way. IMO it would improve the information transfer if the snark was pulled. – John V. Aug 19 at 16:39
  • This answer seems to be a "kitchen sink" approach to a question that's not clear. Indeed, the person asking the question is new to encryption, but I don't think throwing a big wall of text at an unclear problem, and hoping something sticks is the right approach. – Steve Sether Aug 19 at 21:19

If you want perfect forward secrecy (PFS), then one option is is for the sender and recipient to exchange ephemeral ECDHE keys. Then, the message can be encrypted with AES, using the shared secret derived from the ECDHE key exchange as the key. This is how TLS 1.3 does it. However, this requires an additional round trip between the client and the server, for the key exchange.

If PFE is not required, then the sender can just create a random AES key, encrypt the key using the recipient’s public key, then encrypt the message using the AES key, and send both to the recipient. This is essentially how PGP does it. It is simpler than the above because it doesn’t require the additional round for key exchange, but it lacks PFS.

| improve this answer | |
  • 4
    Please don't recommend anyone skips PFS because it's simpler. There are good reasons why the infosec community doesn't recommend pgp. :-) – Z.T. Aug 18 at 18:33
  • @Z.T. I agree. I wanted to make the distinction to show that PFS requires another round for ephemeral key exchange, but I could have clarified that better. – mti2935 Aug 18 at 18:39
  • 1
    "You could implement pedestrian recognition using these neural nets, or you could just drive point to point using GPS, which is simpler than the above but lacks avoidance-of-killing-pedestrians". The TLS 1.3 default 1-RTT handshake has the keyshare in the first message (ClientHello). Signal has keyshares stored on the server to enable offline messages. These problems are solvable, by people like Trevor Perrin, but not by people who want to use "RSA and AES" in 2020. Which mode of operation do you think they mean when they say "AES"? – Z.T. Aug 18 at 18:46
  • True, but for that matter, you could also calculate the value of a European Call option using the Black Scholes model, but that assumes a log-normal distribution of the underlying stock price, and does not account for the possibility of early exercise. Also, don’t forget Moxie. Last but not least, the PGP model does not require any prior action by the recipient (unlike 1-rtf and signal), but again this is just to show contrast to protocols that offer PFS. – mti2935 Aug 18 at 18:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.