I have two network peers that need to use custom, encrypted application protocol to communicate with each other. Exchanged messages are fairly small, usually less than couple of hundreds of bytes, and key exchange is performed in person. Is it then reasonable to make the following assumptions:
No certificate authority is needed, since both parties know each other and agree upon using a particular key pair.
Since the exchanged messages are small, and the amount of traffic very little, encryption and decryption of messages do not cause any measurable spikes in CPU load or network lag, therefore, it is not necessary to use a symmetric session key (all traffic is encrypted using RSA 4096 key pair).
A time stamp, HW ID and HMAC is added to each message to verify both the data integrity and the authentication of the message.
Because of this, full-fledged TLS is not needed, RSA is sufficient.
Am I missing something? Would this mechanism be immune to MITM and replay attacks, as well as brute forcing?