To setup a secure channel with server I use the following setup in my application:

1.Generate a random AES key and random IV

2.Encrypt AES Key, IV and client credentials with RSA public key

3.Send to Server

4.Server decrypts using private RSA key

5.Server verifies credentials

6.Server replies with [(OK or FAIL) + random salt] encrypted with AES key

7.Client verifies response

8.Communication channel is designated as authentic and session as valid

9.All further business communication go via same AES key until session end (Disconnection)

Additional details:

1.Public and private RSA keys are hard-coded and constant for application's lifetime i.e. no procurement of public key from external source happens

2.All constant known plain-texts like OK FAIL etc are sent with salt

I beleive this setup should pretty much be fool proof unless someone has access to client or server operations (or breaks RSA)

Are there any potential loop-holes in this setup that I failed to realise or overlooked?

  • 2
    It looks reasonable, though the devil is always in the details. The thing that comes to mind is.. why re-implement TLS? Why not just use TLS?
    – crovers
    Oct 20, 2016 at 14:56
  • Not using stock TLS because its a socket level bi directional communication instead of http or REST
    – Allahjane
    Oct 20, 2016 at 15:45
  • 1
    @Allahjane TLS isn't specific to HTTP. Also, what block mode are you using for AES? Is it authenticated?
    – Luke Park
    Oct 20, 2016 at 22:06
  • You should use TLS or libsodium, not invent your own buggy inferior protocol. TLS has nothing to do with HTTP. RSA is complicated to implement securely. You probably want a protocol with forward secrecy to now allow decryption when the RSA key leaks. You might even make a mistake in adding MAC to AES if you don't use vetted protocol and implementation.
    – Z.T.
    Oct 29, 2016 at 12:07
  • @Z.T Woah there buddy. Isn't the whole Web running off RSA?
    – Allahjane
    Oct 30, 2016 at 10:01

2 Answers 2


Your concept looks mostly good, but I see two flaws in it:

  1. Not cryptographic authentication: I would suggest that in step 2, instead of sending the credentials you just send a user identifier and derive a MAC key for the user's secret which you use to authenticate the remaining data. This would reduce the possibility of someone tampering with the data and furthermore improve the security by not having to send any secrets.
  2. No forward secrecy: If this is an actual problem depends on your scenario but since you stated that the RSA private key is hard coded in your software, I assume it could be one (especially since your current plan is to send some secret credentials in step 2 always encrypted with the same key). Instead of sending an AES key in step 2, you could send half a Diffie-Hellman key exchange message and return the servers key exchange message in step 6 along with the encrypted response.

Not really a problem but still noteworthy: I do not see a point in creating the IV on the clientside and transfering it encrypted. An IV is not a secret thing, it could just as well be created on the server and sent along with the message in step 6 in cleartext.

  • "RSA private key is hard coded in your software" private key is hardcoded on server and public key on client. the client has no idea about the private key. "This would reduce the possibility of someone tampering with the data" I don't think its possible to change anything in a RSA cipher text by a third party except corrupting it. please correct me if I'm wrong. In case of any error in communication the application is set to reestablish the connection by performing the handshake again
    – Allahjane
    Oct 21, 2016 at 9:43
  • I'm not sure if RSA yields same cipher text for same key and plain-text for the first encryption. As such a random IV in the encrypted frame serves as a salt
    – Allahjane
    Oct 21, 2016 at 9:54
  • Ad authentication: Without authentication, detecting a corruption can be hard. It is more complicated having to parse the user data for corruption than simlpy being able to do a cryptographic integrity check. Especially since the biggest part of your encrypted data is random data anyway. Ad IV: You do not need to salt the plaintext of an RSA encyption as long as you use a good padding like OAEP (what you should do anyway).
    – mat
    Oct 21, 2016 at 10:28

One of the things such a schema should be immune against is MITM and token replay. And here I see a problem. What prevents a MITM to suppress a server's response and replay a message previously captured? Depending on the application payloads you send this can create serious damage.

In order to prevent this you should include a sequence number in your communication. Client sends number X, starting with 1, server reply needs to contain the 2 and so forth. Even better would be to check how TLS addresses this problem and copy this mechanism. Maybe you need to include a timestamp and cache data within one session to get complete replay safetyness.

Also, you do not say what "session end" means. Break of the TCP/IP connection? I think it would be beneficial for security to clearly define how to handle the "session" notion. If a session is pending, is the same client allowed to open a second session for the same user? If the session is idle for X minutes, is it mandatory for the server to resume a session or can it say "no, please establish a new one" (with a new AES session key). Create a possiblity to explicitely close the session through a corresponding message (which allows the partners to free computation resources).

  • A session here indicates the duration in which the client has been authorised after logging in to the moment client wishes to disconnect or an error occurs leading to disconnection
    – Allahjane
    Oct 21, 2016 at 9:45
  • Yeah token replay could be a danger, good catch , since its TCP I think using a 64 bit counter for logical frames would suffice. Apart from that is there anything else a MITM attack do? since the RSA public key is hardcoded there is no way an attacker could change that to become MITM without changing the application code which is beyond the amount of security level of the application design. but even so if there is something that can be done please elaborate
    – Allahjane
    Oct 21, 2016 at 9:51

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .