As part of the end to end encryption for my app, I am following the standard procedure of using RSA to exchange the temporary AES256 key. Currently when A decides the key to send to B, he encrypts it with B's public key. All of this is taking part over a TLS connection to the server. However B is sort of taking the key on blind faith that it came from A. Of course, it's not like anyone can send B the AES256 key. You need to log in first to the server before it's willing to pass messages to A. The AES256 key is also only ever sent during the setup of a session between A and B which B would have to have accepted before. The server will also check that there is a requested session start between X and B before sending anything to B and that it is really X that B wants to have a session with. The point of the end to end is to prevent the server from knowing what's going on between A and B (even if it is my own physical one).

I thought that making A include a signed hash of the key to B would reassure B that the key really did came from A. However, java in android isn't able to encrypt the signed hash because it is exactly 2048 bits (as expected). Would there be any danger in just transmitting the signed hash (Java RSA with SHA512) in plain sight? My understanding is that hashes are brute forced which at this point, you might as well just brute force the AES key anyways.

Edit: Since it seems like people are getting to the heart of the problem: encryption over UDP, what's done is done. I'll write up how I'm currently doing it. It's my own personal project so I welcome any improvement. I'm not emotionally attached to the current method.

(All this take place over an existing TLS connection as the "command" socket is TCP)

  • A generates the AES256 key using java's SecureRandom.
  • A gets a copy of B's public key from the server.
  • A sends an the RSA encryptd key to B using android's RSA/NONE/OAEPWithSHA1AndMGF1Padding. This is chosen for the OAEP padding which (from what I've read) sounds like the best method of making the input to RSA look as random as possible to avoid the output having certain characteristics based on the input.
  • B gets the message and sends an "ok" command to the server which tells A.

(The rest takes place over plain UDP)

  • A sends a message to B using android cipher's AES/GCM/NoPadding. GCM was chosen because (from what I've read) it's the best method for "chopping" up the data into secure pieces. GCM is supposed to contain a MAC so I'm assuming it takes care of authenticating for me. A new cipher instance is instantiated for every block of data. I'm assuming android's cipher.init takes care of the IV (new for every block). After encrypting. the data is sent [IV length;IV;AES256/GCM Data]. From what I've read the IV is like a salt which is used to prevent precalculated attacks. Normally in a database the salt is stored in plain text so in this case it's transmitted in plain. (If it wasn't this creates an infinite regression problem.)
  • B does the reverse process and sends a response to A using the method described above.

A new AES256 key is used for each session between the 2 for forward secrecy. In terms of replay, the plaintext data has a sequence number. I have seen MAC decryption failures in the logs when using LTE at the beginning of the session. Haven't had a plausible explanation why.

  • 1
    Might I ask why not use a standard communication method like TLS on its own?
    – forest
    Mar 18, 2018 at 5:33
  • 2
    I can almost guarantee it is not sufficient. TLS involves a lot of complexities to mitigate a lot of security issues. The protocol is huge. Because OpenSSL works on arbitrary buffers, you could trivially implement TLS over UDP (similar to DTLS) without needing to create your own protocol. I mean, do you use a MAC? Is it ETM or MTE? What block mode do you use? Do you use OAEP padding? Do you use unique IVs? Do you mitigate downgrade attacks? What about replay attacks? Do you have FS? All of these things are dealt with by TLS, but chances are, you haven't mitigated them all yourself.
    – forest
    Mar 18, 2018 at 6:11
  • 2
    I'm confused: You trust the server for a major part of the security here (like proper authentication and maybe also giving A the pubkey of B - a least you don't mention where A gets this key from). But then you do all your complex stuff since you don't trust the server: "The point of the end to end is to prevent the server from knowing what's going on between A and B". In other words: you need to trust the server first in order to avoid trusting it later. Mar 18, 2018 at 6:24
  • 2
    Note that using a new AES key for each session does not give you forward secrecy because you still use RSA. You would need to use something like DHE instead.
    – forest
    Mar 18, 2018 at 13:43
  • 1
    @AAccount Perhaps you should look into libsodium which implements simple and secure key exchange and encryption.
    – forest
    Mar 19, 2018 at 2:05

1 Answer 1


I believe this is a case of the XY problem. You are trying to allow authentication in your protocol scheme, and you are asking how to securely do it using a method which you implicitly assume is necessary. The reality is that there are already plenty of ways to prove that a given key is authentic without sending over a hash of the key. I suggest you read into how TLS provides authentication without actually signing the session key itself. As that's the subject for another question, I'll answer specifically as to whether or not the signed hash of a message (such as a shared key) reveals any information about the contents of the original message (tl;dr the answer is no, it does not).

Signing a message involves computing the hash of the message and signing the hash. Because the signature depends only on the value of the hash, we can safely ignore the signature. It gives away no more information about the message than the hash itself. So can we attack the hash?

An attack that would allow someone to learn information about the input to a hash function given only the digest itself is called a (first-)preimage attack. More specifically, this is an attack against some hash function f where given only digest h, one is able to find an arbitrary message m such that f(m) = h faster than exhaustive search. All modern cryptographic hashes*, such as SHA-512, are conjectured to be preimage-resistant. As such, providing the hash to a message gives away no information about the message itself. The only possible way an attacker could find the message is if it is short enough that it is feasible to brute force the message until a matching hash is found.

For future reference, there are three classes of attacks against a cryptographic hash function f:

  • Preimage attack - Given h where f(m) = h, find any m' such that f(m') = h.

  • 2nd preimage attack - Given m, find any m' such that m ≠ m' and f(m) = f(m').

  • Collision attack - Find any pair of m and m' such that m ≠ m' and f(m) = f(m').

An unbroken cryptographic hash with a digest length of n is assumed to have a first-preimage and second-preimage resistance of 2n and a collision resistance of 2n/2. This is the case for SHA-512.

* Formally, a collision and preimage-resistant cryptographic hash is defined as a function that maps an arbitrary-length input to fixed-length output. For a message space of {0,1}* and some fixed n, the algorithm for cryptographic hash function f is defined as {0,1}* → {0,1}n. This is explained in more detail in an answer on Crypto.SE.

  • While I do plan to look at the TLS link, is there any inherent disadvantage to the signed hash method out of curiosity? But yes, you did get at the heart of the problem.
    – AAccount
    Mar 18, 2018 at 4:55
  • @AAccount If the goal is to verify that whoever possesses the shared key also possesses the (presumably trusted) private signing key, then there are no significant disadvantages security-wise. There are just better solutions out there. For example, why not sign the certificate instead? That way both the shared secret and the MAC key (you are using a MAC, right?) can be trusted.
    – forest
    Mar 18, 2018 at 5:02
  • The current system is setup so that you would have the other person's public key in advance. I could post the entire process I'm using for review which I'd really be interested in getting a second opinion as it's assembled from various sources I've read. Would it be too bothersome to review the process or should I make another question or use the discussion feature?
    – AAccount
    Mar 18, 2018 at 5:38
  • @AAccount Depending on if it's on-topic (i.e. shows more work than just "review my code", for example you explain the assumptions, the guarantees you attempt to provide, etc), then yes. If this answered your specific question then just mark it as solved and create a new question.
    – forest
    Mar 18, 2018 at 5:44
  • I updated the original post with the method of how I'm doing, with why and my assumptions. It's not a code review request in disguise.
    – AAccount
    Mar 18, 2018 at 7:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.