Are there any particular flaws in this protocol?

(Yes I know about SSL)

Three participants:

Login Server LS
Game Server GS
Client C

Asymmetric key-pair Kpriv Kpub
Shared 128bit secret key Ksec
Shared 128bit signing key Ksecsign
Client secret Csec
Login Server nonce NonceLS
Client Login nonce NonceC
Game Server nonce NonceGS

Session S

Before start

LS holds Kpriv, Ksec and Ksecsign
GS hold Ksec, Ksecsign
C holds Kpub

All symmetric encryption uses AES-128, asymmetric uses 2048 bit RSA. HMAC uses SHA-256. PBKDF2 uses HMAC SHA-256.

  1. C connects to LS.
  2. C sends a puzzle request.
  3. LS responds with a puzzle challenge which is a unique random nonce and a difficulty.
  4. C computes a puzzle reply and creates a random nonce Csec
  5. C encrypts nonce + client information (type of device etc) using Kpub.
  6. C Sends a session request with the puzzle reply in the clear, and the rest of the data encrypted (as detailed in 5). I.e. puzzle-reply || EKpub(Csec || device data)
  7. LS queues C's request according to the quality of the puzzle reply. If the queue is full and C's puzzle answer isn't better than the worst reply in the queue, it replies with a request failed. C can then start from (2)
  8. LS decrypts the payload. It then creates NonceLS and creates two keys using PBKDF2 with HmacSHA256 from (NonceLS || Csec) and two different salts.
  9. LS then encrypts (Csec || device data || NonceLS) using Ksec and signs this with a HMAC using Ksecsign. This is the new session. (S <- HMACKsecsign(IV || EKsec(...)) || EKsec(Csec || device data || NonceLS || timestamp) || IV).
  10. LS Then uses the same method (encrypt-then-sign) as above with the derived keys in (8) to create encrypted data to give to the client. I.e. the client receives NonceLS || HMAC(IV || E(S)) || E(S) || IV.
  11. C Construct the same keys as in (8) and use them to verify, then decrypt the message. C now has NonceLS, S and Csec.
  12. C connects to GS
  13. C creates a new nonce NonceC and sends NonceC || S in the clear to GS
  14. GS decrypts and verifies S using the shared Ksec and Ksecsign. It logs the device information (note that forged device information isn't a problem) and creates a new nonce NonceGS. It then generates a key and sign key using different salts with PBKDF2 and NonceC || NonceLS || NonceGS || Csec. (If the session is expired, then an error code is returned, which requires C to request a new S from the LS)
  15. GS Then replies in the clear with (1 byte result || NonceGS)
  16. C Sets up the same key as GS and start sending packets in the following format: (2-byte packet length || HMAC(IV || E(...)) || E(packet number || data) || IV). Packet numbers start at 0 and are increased by 1 for each packet sent. HMAC is first verified, then decryption is done, followed by packet number verification.

For both server and client decryption, if there is any packet integrity error anywhere, the server/client disconnects. In order to reconnect a new handshake is required, which will mean a new NonceC and new NonceGS.

There are some omissions in this protocol as far as I can tell, which I don't know how they will affect the general security of the protocol:

  1. There is no signing in (6)
  2. Packet length can be tampered with.
  3. (10) NonceLS is not included in the HMAC (I will fix this)

I'm also considering switching to - for example - AES in CTR mode for (16) to avoid the IV / packet number / padding overheads for each packet.


I try to assure the following:

  1. Only a server holding Kpriv can issue a valid session S to a client.
  2. No one but the holders of Ksec can decrypt a session S.
  3. The session S is resistant to tampering.
  4. A holder of a session S and the secret key Csec can initiate use these to initiate a confidential connection with a server holding Ksec
  5. Encrypted sessions in (4) is not vulnerable to man-in-the-middle, replay or other attacks.
  6. A large amount of session requests (1) cannot CPU-exchaust a server.
  7. Initiating a encrypted session (4) is inexpensive CPU-wise.
  8. Session requests and replies are not vulnerable to attacks such as replay or man-in-the-middle.


To clarify, this is TCP all the way.


The key exchange works like this:

Client A, Login Server S, Game Server B

(1) A → S : { Na, A }Kpub

Ksa ← { Na, Ns, A, Saltenc }PBKDF2

Ksasign ← { Na, Ns, A, Saltsign }PBKDF2

M ← IV || HMAC(IV, { Na, A }Ksb)Ksbsign || { Na, A }Ksb

(2) S ➝ A : { Ns, IV, HMAC(IV, { Ns, M }Ksa)Ksasign, { Ns, M }Ksa }

(3) A ➝ B : { N'a, M }

(4) B ➝ A : { Nb }

Kab ← { Na, N'a Nb, Saltenc }PBKDF2

Kabsign ← { Na, N'a Nb, Saltsign }PBKDF2

Encryption proceeds using Kab, signing each with Kabsign and keeping a running counter with each packet.

There are some weaknesses in this as written.

If you have a well-tested strong key exchange protocol I could use as a drop-in replacement that would be fine.

The requirements are:

  1. No communication between S and B.
  2. S and B can share a finite limits of secrets before all exchanges.
  3. A possess a public key and S holds the private counterpart.
  4. A-B key negotiation should not use asymmetric key exchange.
  • 9
    What are you trying to ensure with your protocol? Without that information, there can be no flaws.
    – Ladadadada
    Jun 17, 2013 at 8:52
  • @Ladadadada updated the question.
    – Nuoji
    Jun 17, 2013 at 9:28
  • 1
    My initial reaction is that this protocol is inefficient, but that is a more difficult problem to fix. Are you using AES in CTR mode or another mode? You should have committed to a mode of operation before talking about the protocol. You need integrity checking on every encrypted request (or you could construct a cryptography oracle...), step 6 can be signed with kpub. Also what is the "puzzle"?
    – rook
    Jun 17, 2013 at 16:56
  • 1
    @Rook inefficient in what sense? I was originally simply running AES CBC, which I believe should be ok against attacks as long as the HMAC check is correct. The puzzle is just find a value n such that SHA1(challenge nonce | n) has at least <difficulty> number of trailing zeroes. Integrity checks are everywhere except for the initial public key. I'll add it, but is there really room for an exploit here? Is it really helpful to alter the public key-signed request?
    – Nuoji
    Jun 18, 2013 at 0:05
  • @Nuoji well first off, using key stretching with PBKDF2 in an HMAC with a secret as a derived key? The threat of DoS is is greater than the threat posed by brute forcing such a massive keyspace in a short lived session. What advantage does this HMAC have over an authenticated mode like GCM or CCM? Authenticated modes lend them selves to beatiful designs. In terms of the second question: at no point should you ever have secret transmitted over an insecure channel without verifying that this secret wasn't modified. Padding oracles and decryption oracles affect AES-CBC. But still a concern.
    – rook
    Jun 18, 2013 at 0:45

1 Answer 1


Note: according to comments, this protocol doesn't use UDP. But i'll leave the answer here as it might be useful to someone else.

From your language it sounds like you're using UDP instead of TCP. Perhaps you're not, but you didn't specify. And if you are, then here's something to think about.

This is especially significant if the packet boundaries are not random and unpredictable.

If you're ignoring dropped packets (as almost all UDP apps do), see the paragraph below. If you aren't, then you'll have to think about retransmission (and when and how to trigger it), out-of-order arrival, sliding window oriented reassembly, window size and expansion, and path MTU and MTU discovery... and/or fragmentation. I guess that applies even if you don't discard packets. This can really affect network throughput, especially given the fact that arrival order is significant from a security standpoint. In other words, if you're not ignoring dropped packets, the consider TCP instead.

But if you do discard dropped packets, then the size of the packets (or timing, or other similar metadata) may be indicative of the action that the packet represents. That's an information disclosure vulnerability already (exploitable? depends on context). But if a MITM could selectively discard or otherwise affect packets based on their function, that could trigger a false assumption in the client or server. Perhaps a transaction is confirmed only to one party, perhaps certain actions or commands can be filtered, perhaps some security-related function can be suppressed.

Again, it all depends on context. Which I guess is just another way of saying that there's not enough information to evaluate this properly.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.