Security for a low latency online multiplayer game

Question

I am designing an online multiplayer game and I am looking for a good tradeoff of secure communications with minimal CPU and bandwidth utilization. My ideal solution would only use UDP packets since TCP is a poor choice for the realtime requirement of the game. However, I have no problem falling back on TCP for the login server.

Here is my idea.

---

When users first login to the service:

1. Users connect to an authentication server (which holds a Verisign cerficate) via SSL.
2. The authentication server validates the user's login credentials.
3. If the login is valid, then two 256-bit AES public/private key pairs are generated by the server. The user is sent one private key and one public key to remember, which can later be used for symmetric communication. The server remembers the opposing key pairs.
4. Additionally, a random 32-bit integer is generated and sent to the user. I will call this the "obfuscation key" for now.
5. The SSL connection is ended. From here on, all of the data is sent through UDP.

Whenever a user needs to perform a secure transaction with a server:

1. For requests, the payload is encrypted using the public key that the client received.
2. For responses, the payload is decrypted using the private key that the client received.

My first question -- is this safe and advisable?

---

The other problem I am trying to solve is sending timely packets for updates that don't have to be as secure but are extremely frequent, such as player movement and position.

Sending the data in the clear is dangerous due to trivial packet replay and man-in-the-middle attacks. However, I believe that performing the AES encryption for every such packet would be too CPU intensive. Instead, I am considering a simple XOR cipher obfuscation as follows:

1. Create a key for the XOR cipher by XOR-ing a 32-bit timestamp with the "obfuscation key" received earlier.
2. Apply the XOR cipher to the payload of the UDP packet.
3. Send the UDP packet with the timestamp in the header plus the obfuscated payload. This way the server can de-obfuscate using the same timestamp.

One thing I hope to accomplish with this is to simply deter packet sniffers and "script kiddies" who might try to trivially reverse engineer the protocol. Another concern is preventing packet replay attacks, which is the reason for including the timestamp in the XOR cipher. Finally, I'm hoping that adding the "obfuscation key" into the mix will prevent trivial man-in-the-middle attacks.

My second question -- is there a simpler and/or more effective way to hit these goals?

I realize these are loaded questions, so thank you in advance for taking the time to read this.

Answer 1

To the extent that I understand what you are proposing, I'm concerned this is not a great design. It feels like it might be more complex than necessarily. I'm concerned by any design that has the server knowing the client's private keys; feels like an unnecessary risk. The use of "obfuscation keys", XOR ciphers, and the like raises red flags. For these reasons, I would have concerns about its safety.

I'm wondering if there is a simpler solution. My recommendation would be: check out Datagram Transport Layer Security (DTLS). DTLS is a variant of TLS designed especially for datagram protocols, like UDP. It is designed to work well even in the presence of dropped and re-ordered packets. DTLS is reasonably well-vetted. Using an accepted, published, well-vetted scheme is much safer (and much cheaper) than designing your own. Therefore, if DTLS meets your needs, it seems like it would be a great solution.

If it doesn't meet your needs, come back and let us know what requirements it does not meet, and we may be able to make some other suggestion.