What are the downsides of not adhering to TLS, eg, directly using encryption algorithms (2048-bit RSA, 128-bit AES both with proper padding) provided by SSL library to generate keys, encrypt, use proprietary "certificates" and proprietary protocol in general. First thing that comes to mind is that this will prevent downgrade attacks, and allow developer to implement mechanism against timing attacks, my main reasoning here is that to be sure that library won't do anything dangerous behind the scenes.
closed as unclear what you're asking by Steffen Ullrich, Xiong Chiamiov, WhiteWinterWolf, TheJulyPlot, Serge Ballesta Jul 20 '17 at 9:43
Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.
The biggest downside is that you're expecting that because you're writing something smaller, it will also be safer. In reality, this is really hard to do.
Given that you must control both ends of the connection in order to even consider implementing a custom protocol in place of TLS, there are other way you can increase the security and reduce the attack surface without going through the exercise of inventing and implementing a custom protocol and hoping it's safer.
You want to prevent downgrade attacks? Disable everything except for TLS v1.2 on your servers and clients.
You want to ensure only strong cipher suites are used? Disable anything except for your AEAD cipher suites.
You have lots of options when using existing TLS libraries to lock them down and harden them, and even with all of their warts, they've undergone far more analysis than a custom protocol you build ever would. You're better off not re-implementing the wheel, but instead simply taking advantage of your opportunity to configure TLS as securely as possible.