But what's about rolling your own (encryption supporting) network protocol?
I'm working on an open-source project (by now it's an idea to be honest) where I need to exchange a lot of binary data between clients, but transfering text information with the data or even alone is an important aspect of inter-node communication and having two seperate protocols for doing so is an outright tdwtf.
My basic idea would be to write my own protocol specification with header, text- and bin-section and encrypt all of this with RSA (the question is not about reviewing this idea).
Is there anything bad in writing my own protocol security-wise* instead of relying on something existing, yet maybe not fitting?
* I do see that this maybe will make it harder for third-party clients and that this is not a security boundary.
EDIT: To clarify, I'm talking about a protocol on top of TCP (like HTTP). The question is about the security implications when I'm rolling my own protocol with a well known cipher integrated.