My application uses UDP. Typical Client-Server Communication consists of one packet (< 500 bytes payload) requests from the client and one packet answers to those requests from the server. In the case of packet loss the client has to resend its request.
To secure the "connection" an AES-128 PSK is used (previously transferred via TLS/SSL). Now each packet will be encrypted using AES-128 in CTR mode. The packet will contain the nonce/counter, the message to be delivered and a hash of the nonce plus message:
MyPayload = aes128CTR(nonce + plaintext + hash(nonce + plaintext))
Can a non cryptographic hash function (e.g. Murmur3) be used as a MAC in this case?
Furthermore do I have to send the complete nonce (16 bytes) or will a smaller counter (e.g. 4 bytes) be enough to be sure the packet was indeed sent by my client instead of being a possibly IP-spoofed packet?