This is probably a naive question, but here goes: What is better for security over TCP/IP?

  1. Encrypting the payload and then adding a checksum?
  2. Checksuming the payload and then encrypting it?
  3. Checksuming the payload and then encrypting it, then checksuming again?
  4. Something else entirely?

And by security, I mean bad guys can't read the payload, replay the messages, inject their own messages, or modify the messages in transit without detection.

  • I wish I could mark more than one answer correct, as all the answers were very helpful. @Terry Chia's link ended up being the most informative to me. – kmort Apr 15 '13 at 13:34

I think you mean using a MAC when you say checksum.

The ideal situation is encrypt-then-MAC. See this question for more information why.


For the practical answer, what's best for security is to avoid reinventing the wheel. Designing a secure protocol for data transport is a daunting task (it's easy to make it work, it is hard to know whether it is "secure"). There are already a few known protocols which have been studied a lot, and are available as ready to use library. Foremost of which is SSL/TLS. You might want to use DTLS if you want to protect individual packets (DTLS is to UDP what TLS is to TCP).

For the scientific answer: when it comes to encrypting some data and protecting its integrity, best idea is to use an encryption mode which does both (GCM and EAX are the usual suggestions). If you have to rely on separate algorithms for the encryption and for the integrity check (what you apparently call a "checksum" but is better known as a MAC), then the correct way is to encrypt first, then compute the MAC on the encrypted data (not forgetting the IV and some other stuff, as explained in this question).

But let me point out that this is only part of the protocol. Doing encryption and a MAC on a "payload" assumes that sender and receiver already share a common high-entropy secret value which they can use as key for the encryption and MAC. You also need some more features to avoid replay attacks, and, more generally, defeat attackers who would try to drop packets, duplicate packets, and reorder packets. Authentication (one-way or mutual) is yet another tricky area.


It sounds like you are reinventing "Tunnel mode ESP" (which is the only useful protocol in IPSec https://en.wikipedia.org/wiki/IPsec)

It encrypts then signs.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.