I am considering how to make my chat application a bit more secure. It has a similar structure to IRC, multiple users can join each server, and servers can be hosted by anyone. I have been using TLS so far, but it has the problem of requiring each server to have a signed certificate, which is not very practical because it is important that hosting a server be as easy as possible and signed certificates often cost money. I was considering using SSH, but it seemed rather difficult to use it purely as an encrypted wrapper over a TCP socket. Additionally I need to be able to connect via websockets from a browser (so that I can run a client in the browser), and therefore I'd have to find some way of implementing the SSH protocol inside the web browser which seems impractical.
Security is not the number one priority for this program, but it needs to be safe from a MITM attack because passwords are sent over the connection. Of course the person running the server could also intercept these passwords but I don't think there's much that can be done about that.
Should I just use TLS and tell the users to get a signed certificate, or are there any other protocols that I could use instead to wrap a TCP (or websocket) connection? Is it even somewhat feasible to implement a custom protocol (of course using existing crypto libraries)?