As part of the end to end encryption for my app, I am following the standard procedure of using RSA to exchange the temporary AES256 key. Currently when A decides the key to send to B, he encrypts it with B's public key. All of this is taking part over a TLS connection to the server. However B is sort of taking the key on blind faith that it came from A. Of course, it's not like anyone can send B the AES256 key. You need to log in first to the server before it's willing to pass messages to A. The AES256 key is also only ever sent during the setup of a session between A and B which B would have to have accepted before. The server will also check that there is a requested session start between X and B before sending anything to B and that it is really X that B wants to have a session with. The point of the end to end is to prevent the server from knowing what's going on between A and B (even if it is my own physical one).
I thought that making A include a signed hash of the key to B would reassure B that the key really did came from A. However, java in android isn't able to encrypt the signed hash because it is exactly 2048 bits (as expected). Would there be any danger in just transmitting the signed hash (Java RSA with SHA512) in plain sight? My understanding is that hashes are brute forced which at this point, you might as well just brute force the AES key anyways.
Edit: Since it seems like people are getting to the heart of the problem: encryption over UDP, what's done is done. I'll write up how I'm currently doing it. It's my own personal project so I welcome any improvement. I'm not emotionally attached to the current method.
(All this take place over an existing TLS connection as the "command" socket is TCP)
- A generates the AES256 key using java's SecureRandom.
- A gets a copy of B's public key from the server.
- A sends an the RSA encryptd key to B using android's RSA/NONE/OAEPWithSHA1AndMGF1Padding. This is chosen for the OAEP padding which (from what I've read) sounds like the best method of making the input to RSA look as random as possible to avoid the output having certain characteristics based on the input.
- B gets the message and sends an "ok" command to the server which tells A.
(The rest takes place over plain UDP)
- A sends a message to B using android cipher's AES/GCM/NoPadding. GCM was chosen because (from what I've read) it's the best method for "chopping" up the data into secure pieces. GCM is supposed to contain a MAC so I'm assuming it takes care of authenticating for me. A new cipher instance is instantiated for every block of data. I'm assuming android's cipher.init takes care of the IV (new for every block). After encrypting. the data is sent [IV length;IV;AES256/GCM Data]. From what I've read the IV is like a salt which is used to prevent precalculated attacks. Normally in a database the salt is stored in plain text so in this case it's transmitted in plain. (If it wasn't this creates an infinite regression problem.)
- B does the reverse process and sends a response to A using the method described above.
A new AES256 key is used for each session between the 2 for forward secrecy. In terms of replay, the plaintext data has a sequence number. I have seen MAC decryption failures in the logs when using LTE at the beginning of the session. Haven't had a plausible explanation why.