By now, everyone on here should know that rolling your own crypto is bad.

But what's about rolling your own (encryption supporting) network protocol?

I'm working on an open-source project (by now it's an idea to be honest) where I need to exchange a lot of binary data between clients, but transfering text information with the data or even alone is an important aspect of inter-node communication and having two seperate protocols for doing so is an outright tdwtf.

My basic idea would be to write my own protocol specification with header, text- and bin-section and encrypt all of this with RSA (the question is not about reviewing this idea).

Is there anything bad in writing my own protocol security-wise* instead of relying on something existing, yet maybe not fitting?

* I do see that this maybe will make it harder for third-party clients and that this is not a security boundary.

EDIT: To clarify, I'm talking about a protocol on top of TCP (like HTTP). The question is about the security implications when I'm rolling my own protocol with a well known cipher integrated.

  • I'm talking about a protocol like HTTP, so on the TCP layer. The linked questions are about rolling your own crypto, which is making your own way of encrypting things. I'm asking about making a own protocol like http, which would best fit my programs needs and the security implications of this (when using a known cipher). – Sebb Oct 2 '15 at 7:13
  • Have you looked at all the variants of TLS/SSL and found them to be not suitable for securing you custom protocol? – jhash Oct 2 '15 at 11:11

Having pentested a number of custom network protocols (all implemented on top of TCP or UDP, sometimes with (D)TLS as well), I recommend against this idea. You will get it wrong, and then you'll have a remote exploit vector.

It is much, much safer to use an existing data format (JSON via HTTP is popular these days, though yes, that's a lot of overhead) with established server and client software, established parsing libraries for your data representation, and a fairly simple format (don't get fancy unless performance actually, sincerely, requires it). Use TLS (or DTLS if using UDP) to secure the traffic. Add your own authentication if required.

Not the answer you're looking for? Browse other questions tagged or ask your own question.