I'm working with a program with a networking protocol that is using RSA keys to authenticate clients. I am looking to replace the custom authentication protocol with TLS (as it's a vetted protocol). I can use self-signed certificates or RFC 7250 for client authentication. The problem is that there are some use cases for the program where the standard PKI system (i.e. trusted root certificates) isn't viable for authenticating the server. (The program needs to be able to connect to servers on a LAN and it's not reasonable to have users setup their own roots to do it). Looking into "client-only authentication", I found questions like this one where the answer suggests reversing the TLS client and server roles. However, I'm not sure that's the right answer for my use case. I could also just have the server use its own cert/key as its identity and have the client accept any identity when it doesn't know what to expect. I have problems/questions about both solutions:
Using the server identities seems like the most sensible solution to me. Most connections will be able to use an authority to know what the server identity should be, the one case where an un-authenticated client needs to be able to connect will be simpler, and using the protocol in the correct direction might have some optimization advantages. The issue is that I don't know if client authentication is secure when the server authentication may not be. Looking at the specifications, the client and server authentication protocols are similar enough that it should still be secure, but since the spec doesn't allow for "client-only authentication", it's clearly not an intended use case. Is there any reason/evidence to believe that client authentication would or would not be secure when ignoring the server's authentication identity?
Having the client act as the TLS server, being a method suggested multiple times here, seems to be the method with the most guarunteed security, but would be a little tricker to implement. The main thing I'm interested in with this method is what other effects of switching the TLS roles would be. TLS is obviously an asymmetric protocol, but what other asymmetries are there other than authentication? The one I know of is that, while the server ultimately chooses the cipher suite, it's the client that indicates its cipher suite preference that the server normally considers. I found
SSL_OP_CIPHER_SERVER_PREFERENCE to correct this in OpenSSL. What other effects of switching the TLS roles should I consider?
Which method would you reccomend in the end? To add a few points, I'm not worried about the server sometimes not being authenticated. Confidentiality isn't a high priority (just a side effect of authentication), and the protocol is mostly the client sending "commands" to the server, so someone impersonating a server isn't going to be able to do much. The only thing that server authentication could reasonably protect would be an optional password authentication, but I'm looking into implementing a PAKE for that, so it shouldn't be a concern.