I’ve designed a custom network protocol using high-level LibSodium primitives for authentication and encryption. The purpose of the protocol is for a game, and the security is to ensure only paid players can connect and play. I’m offloading initial authentication to a webserver over HTTPS, so my protocol is merely transferring the client from the HTTPS connection with the webserver to a connection on a backend dedicated game server. It is important to ensure only authenticated clients are in the game servers to save money on server runtime costs. No sensitive personal data is ever touched or seen by the client or dedicated servers — the only risk is server runtime.
I’m now at the point of looking for a way to verify the security of the design. I’ve done my best to research many kinds of attacks, such as DoS, MitM, replay, reflection, and ip spoofing attacks.
What kind of steps would you recommend, beyond self-study and self-verification? The protocol is fairly simple, but perhaps a little too long to post here directly.
Here were some pragmatic ideas I had:
- Visit local university to find security professors who are interested in answering questions.
- I work at Microsoft - I could potentially look to make friends with security engineers here.
- Third party security audit - I have no idea if these are respectable or even affordable.
- Simply release the game in the wild - I have low-risk in the event of security compromises, and can potentially patch vulnerabilities whenever they are detected from actual attackers.
How would you go about verifying the design of a custom protocol with similar use-cases?