For the practical answer, what's best for security is to avoid reinventing the wheel. Designing a secure protocol for data transport is a daunting task (it's easy to make it work, it is hard to know whether it is "secure"). There are already a few known protocols which have been studied a lot, and are available as ready to use library. Foremost of which is SSL/TLS. You might want to use DTLS if you want to protect individual packets (DTLS is to UDP what TLS is to TCP).
For the scientific answer: when it comes to encrypting some data and protecting its integrity, best idea is to use an encryption mode which does both (GCM and EAX are the usual suggestions). If you have to rely on separate algorithms for the encryption and for the integrity check (what you apparently call a "checksum" but is better known as a MAC), then the correct way is to encrypt first, then compute the MAC on the encrypted data (not forgetting the IV and some other stuff, as explained in this question).
But let me point out that this is only part of the protocol. Doing encryption and a MAC on a "payload" assumes that sender and receiver already share a common high-entropy secret value which they can use as key for the encryption and MAC. You also need some more features to avoid replay attacks, and, more generally, defeat attackers who would try to drop packets, duplicate packets, and reorder packets. Authentication (one-way or mutual) is yet another tricky area.