Are there any flaws in two step (RSA + AES) key exchange and secure channel setup?

Question

To setup a secure channel with server I use the following setup in my application:

1.Generate a random AES key and random IV

2.Encrypt AES Key, IV and client credentials with RSA public key

3.Send to Server

4.Server decrypts using private RSA key

5.Server verifies credentials

6.Server replies with [(OK or FAIL) + random salt] encrypted with AES key

7.Client verifies response

8.Communication channel is designated as authentic and session as valid

9.All further business communication go via same AES key until session end (Disconnection)

Additional details:

1.Public and private RSA keys are hard-coded and constant for application's lifetime i.e. no procurement of public key from external source happens

2.All constant known plain-texts like OK FAIL etc are sent with salt

I beleive this setup should pretty much be fool proof unless someone has access to client or server operations (or breaks RSA)

Are there any potential loop-holes in this setup that I failed to realise or overlooked?

Answer 1

Your concept looks mostly good, but I see two flaws in it:

1. Not cryptographic authentication: I would suggest that in step 2, instead of sending the credentials you just send a user identifier and derive a MAC key for the user's secret which you use to authenticate the remaining data. This would reduce the possibility of someone tampering with the data and furthermore improve the security by not having to send any secrets.
2. No forward secrecy: If this is an actual problem depends on your scenario but since you stated that the RSA private key is hard coded in your software, I assume it could be one (especially since your current plan is to send some secret credentials in step 2 always encrypted with the same key). Instead of sending an AES key in step 2, you could send half a Diffie-Hellman key exchange message and return the servers key exchange message in step 6 along with the encrypted response.

Not really a problem but still noteworthy: I do not see a point in creating the IV on the clientside and transfering it encrypted. An IV is not a secret thing, it could just as well be created on the server and sent along with the message in step 6 in cleartext.

Answer 2

One of the things such a schema should be immune against is MITM and token replay. And here I see a problem. What prevents a MITM to suppress a server's response and replay a message previously captured? Depending on the application payloads you send this can create serious damage.

In order to prevent this you should include a sequence number in your communication. Client sends number X, starting with 1, server reply needs to contain the 2 and so forth. Even better would be to check how TLS addresses this problem and copy this mechanism. Maybe you need to include a timestamp and cache data within one session to get complete replay safetyness.

Also, you do not say what "session end" means. Break of the TCP/IP connection? I think it would be beneficial for security to clearly define how to handle the "session" notion. If a session is pending, is the same client allowed to open a second session for the same user? If the session is idle for X minutes, is it mandatory for the server to resume a session or can it say "no, please establish a new one" (with a new AES session key). Create a possiblity to explicitely close the session through a corresponding message (which allows the partners to free computation resources).