Does a signed hash reveal any information about the original message?

Question

As part of the end to end encryption for my app, I am following the standard procedure of using RSA to exchange the temporary AES256 key. Currently when A decides the key to send to B, he encrypts it with B's public key. All of this is taking part over a TLS connection to the server. However B is sort of taking the key on blind faith that it came from A. Of course, it's not like anyone can send B the AES256 key. You need to log in first to the server before it's willing to pass messages to A. The AES256 key is also only ever sent during the setup of a session between A and B which B would have to have accepted before. The server will also check that there is a requested session start between X and B before sending anything to B and that it is really X that B wants to have a session with. The point of the end to end is to prevent the server from knowing what's going on between A and B (even if it is my own physical one).

I thought that making A include a signed hash of the key to B would reassure B that the key really did came from A. However, java in android isn't able to encrypt the signed hash because it is exactly 2048 bits (as expected). Would there be any danger in just transmitting the signed hash (Java RSA with SHA512) in plain sight? My understanding is that hashes are brute forced which at this point, you might as well just brute force the AES key anyways.

Edit: Since it seems like people are getting to the heart of the problem: encryption over UDP, what's done is done. I'll write up how I'm currently doing it. It's my own personal project so I welcome any improvement. I'm not emotionally attached to the current method.

(All this take place over an existing TLS connection as the "command" socket is TCP)

• A generates the AES256 key using java's SecureRandom.
• A gets a copy of B's public key from the server.
• A sends an the RSA encryptd key to B using android's RSA/NONE/OAEPWithSHA1AndMGF1Padding. This is chosen for the OAEP padding which (from what I've read) sounds like the best method of making the input to RSA look as random as possible to avoid the output having certain characteristics based on the input.
• B gets the message and sends an "ok" command to the server which tells A.

(The rest takes place over plain UDP)

• A sends a message to B using android cipher's AES/GCM/NoPadding. GCM was chosen because (from what I've read) it's the best method for "chopping" up the data into secure pieces. GCM is supposed to contain a MAC so I'm assuming it takes care of authenticating for me. A new cipher instance is instantiated for every block of data. I'm assuming android's cipher.init takes care of the IV (new for every block). After encrypting. the data is sent [IV length;IV;AES256/GCM Data]. From what I've read the IV is like a salt which is used to prevent precalculated attacks. Normally in a database the salt is stored in plain text so in this case it's transmitted in plain. (If it wasn't this creates an infinite regression problem.)
• B does the reverse process and sends a response to A using the method described above.

A new AES256 key is used for each session between the 2 for forward secrecy. In terms of replay, the plaintext data has a sequence number. I have seen MAC decryption failures in the logs when using LTE at the beginning of the session. Haven't had a plausible explanation why.

Answer 1

I believe this is a case of the XY problem. You are trying to allow authentication in your protocol scheme, and you are asking how to securely do it using a method which you implicitly assume is necessary. The reality is that there are already plenty of ways to prove that a given key is authentic without sending over a hash of the key. I suggest you read into how TLS provides authentication without actually signing the session key itself. As that's the subject for another question, I'll answer specifically as to whether or not the signed hash of a message (such as a shared key) reveals any information about the contents of the original message (tl;dr the answer is no, it does not).

Signing a message involves computing the hash of the message and signing the hash. Because the signature depends only on the value of the hash, we can safely ignore the signature. It gives away no more information about the message than the hash itself. So can we attack the hash?

An attack that would allow someone to learn information about the input to a hash function given only the digest itself is called a (first-)preimage attack. More specifically, this is an attack against some hash function f where given only digest h, one is able to find an arbitrary message m such that f(m) = h faster than exhaustive search. All modern cryptographic hashes^*, such as SHA-512, are conjectured to be preimage-resistant. As such, providing the hash to a message gives away no information about the message itself. The only possible way an attacker could find the message is if it is short enough that it is feasible to brute force the message until a matching hash is found.

For future reference, there are three classes of attacks against a cryptographic hash function f:

• Preimage attack - Given h where f(m) = h, find any m' such that f(m') = h.

• 2nd preimage attack - Given m, find any m' such that m ≠ m' and f(m) = f(m').

• Collision attack - Find any pair of m and m' such that m ≠ m' and f(m) = f(m').

An unbroken cryptographic hash with a digest length of n is assumed to have a first-preimage and second-preimage resistance of 2ⁿ and a collision resistance of 2^n/2. This is the case for SHA-512.

_{* Formally, a collision and preimage-resistant cryptographic hash is defined as a function that maps an arbitrary-length input to fixed-length output. For a message space of {0,1}^* and some fixed n, the algorithm for cryptographic hash function f is defined as {0,1}^* → {0,1}ⁿ. This is explained in more detail in an answer on Crypto.SE.}