I have two network peers that need to use custom, encrypted application protocol to communicate with each other. Exchanged messages are fairly small, usually less than couple of hundreds of bytes, and key exchange is performed in person. Is it then reasonable to make the following assumptions:

  1. No certificate authority is needed, since both parties know each other and agree upon using a particular key pair.

  2. Since the exchanged messages are small, and the amount of traffic very little, encryption and decryption of messages do not cause any measurable spikes in CPU load or network lag, therefore, it is not necessary to use a symmetric session key (all traffic is encrypted using RSA 4096 key pair).

  3. A time stamp, HW ID and HMAC is added to each message to verify both the data integrity and the authentication of the message.

  4. Because of this, full-fledged TLS is not needed, RSA is sufficient.

Am I missing something? Would this mechanism be immune to MITM and replay attacks, as well as brute forcing?

1 Answer 1


all traffic is encrypted using RSA 4096 key pair

So you don't like performance?


Wait, HMAC — that requires a shared key. If you have a shared authentication key, why are you using RSA at all? Use a shared key for confidentiality as well.

A time stamp

What if the clocks diverge?

Because of this, full-fledged TLS is not needed, RSA is sufficient.

You've got it backwards. What, in your scenario, requires using a non-standard protocol? A non-standard protocol means:

  • More bugs because it hasn't had as much review by experts.
  • More bugs because you had to write custom code instead of using well-tested, maintained libraries.

Note that I haven't reviewed your protocol, 1. because you didn't tell us what your protocol is, and 2. because reviewing a protocol is rather more work than expected in a Stack Exchange answer. But just spending one minute with the things you mentioned raises several red-orange flags.

Use TLS with a pre-shared key unless you have a good reason to do otherwise. One good reason not to use TLS is if you want to exchange individual messages rather than maintain a session. In this case, the round trips necessary for TLS are a waste, and you should probably use DTLS instead.

  • I agree with you, I wasn't specific enough. Let me rephrase the question - which one of the two is more secure from standpoint of packet inspection and which one is more CPU intensive: 1. A few very short messages are asymmetrically encrypted using key pairs, the integrity of each message is verified with an HMAC that uses a symmetric key. Both symmetric and asymmetric keys are exchanged offline. 2. Asymmetric encryption using key pairs (exchanged offline) is used only to transfer a symmetric session key, which is in turn used to encrypt a few short messages and ensure their integrity.
    – Ulrik
    Mar 2, 2017 at 12:47
  • @Ulrik If you have a shared key, why do you use asymmetric cryptography at all? Mar 2, 2017 at 16:22
  • If we have one server and two or three clients, those clients can decrypt messages coming only from the server. and no one can impersonate the server.
    – Ulrik
    Mar 2, 2017 at 16:49
  • @Ulrik Uh? Anyone who can validate HMAC's can generate valid HMAC's. So with your protocol, any participant can impersonate the server. Mar 2, 2017 at 22:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.