3

My application uses UDP. Typical Client-Server Communication consists of one packet (< 500 bytes payload) requests from the client and one packet answers to those requests from the server. In the case of packet loss the client has to resend its request.

To secure the "connection" an AES-128 PSK is used (previously transferred via TLS/SSL). Now each packet will be encrypted using AES-128 in CTR mode. The packet will contain the nonce/counter, the message to be delivered and a hash of the nonce plus message:

MyPayload = aes128CTR(nonce + plaintext + hash(nonce + plaintext))

Can a non cryptographic hash function (e.g. Murmur3) be used as a MAC in this case?

Furthermore do I have to send the complete nonce (16 bytes) or will a smaller counter (e.g. 4 bytes) be enough to be sure the packet was indeed sent by my client instead of being a possibly IP-spoofed packet?

  • Where is the non-cryptographic hash being used? Is it the hash in the MyPayload portion (that is encrypted), or are you computing the MAC on the ciphertext? – mikeazo Sep 1 '16 at 12:35
  • 1
    Also of interest: Should we MAC-then-encrypt or encrypt-then-MAC?. – billc.cn Sep 1 '16 at 12:44
  • @mikeazo it's indeed used in the hash within the MyPayloadportion. – HannesR-O Sep 1 '16 at 17:08
3

I suppose not, because the hash serves as a bias to your plaintext. Murmur3, DJB2 etc have very small fixed domains and take collisions into account, which you cannot allow on a counter mode block cipher.

Assume an adversary is aware of the symmetric encryption parameters and block mode operation used, it would only take the size of the nonce + hash domain to decrypt the ciphertext. This can be done in polynomial time. This also answers your second question regarding the counter size.

I would argue using a hash function the way you describe can weaken the ciphertext even further since you derive a relative small hash from the plaintext. I would suggest encrypt-then-MAC either way.

Update

If you have the computational power to establish a TLS session anyway, why not go with a GCM/GHASH variant? You might even want to chain the packets together and add the packet counter as AEAD in GCM.

  • thanks, I'm fairly new to this whole stuff. GCM seems to be exactly what I'm looking for. – HannesR-O Sep 1 '16 at 17:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.