Custom security on top of TCP for a small server

Question

Suppose I want a server to provide some simple, primarily file related functionality and I want to allow access to all users that have a certain public key (server has the private key) and password to the server. It would be a server that few people would know the purpose for (IP would be explicitly shared only with a handful of people), and all its ports besides the one for this functionality would be non-listening. None of the data is important, the main goal is to just avoid any potential issue with someone attacking the server for the heck of it.

What would be the issues/weaknesses with the following approach? I'm not really interested whether it's inefficient, just what the security risks are if I can assume an attacker does not have access to the public key, private key, or the password.

1. Client encrypts password with public key and sends it over to the server. Server decrypts it, validates the password, or TCP session ends. The password is distributed to users, and not possible for users to assign, so hashing it does not seem crucial.
2. Client generates a symmetric key (AES), encrypts it with public key and sends it over to server.
3. Server creates a nonce, encrypts it with symmetric key, and sends it to client. Client decrypts it, concatenates it with a string T (hardcoded constant known to clients and server), encrypts it, and sends to server. Server validates that it received nonce + T, else session ends.
4. Data transfer starts, using the established symmetric key.

I can see there's no authentication of the server - but I can't really any issue with that.

Answer 1

1. Sure.
2. Okay. The acceptance of both these points relies on me choking on the gut reaction to write "We don't encrypt with asymmetric keys directly."
3. I don't really understand this one. The client is the one that sent the key to you, not sure why it's necessary to do this challenge and response. If T is some sort of secret, how does it differentiate from the password in step 1?
4. Fine

HOWEVER

This has ZERO forward secrecy. Compromise of the server private key will enable decryption of all communications, even at a later date. Compromise of the symmetric key by an active attacker in step 2 will lead to decryption of step 4.

As mentioned, writing your own secure protocols is kinda a big no-no. If TLS is a bit much for you, I'd suggest looking at the Noise protocol. The NKPsk0 pattern seems to be what you want (server is known by their asymmetric, clients are known by their preshared secret).

Answer 2

There are two potential downsides that strike me here:

1. Complexity. This is a fairly complex arrangement, I am assuming that you control the software that does the actual crypto on the client side but even so, in my experience this sort of thing tends to break all the time, and eventually your users get fed up with you and go find some other solution. (unless the user is just you, in which case you just end up spending way more time than necessary fixing it.)

2. You're rolling your own crypto here, to a certain extent, which is generally discouraged because it's very easy to make subtle mistakes that don't break anything (in the sense of making it unusable), but make your whole arrangement useless. For example, I don't recall the precise details but a while back there was a story about a widely used implementation of some hashing algorithm (possibly bcrypt?) that assumed a null byte indicated the end of a string, so if you passed it a sequence with a null byte it would disregard everything following. Eventually, someone decided to pass it the output of a different hashing algorithm, I think SHA256, so that it would be possible to use arbitrary-length inputs. Only problem is, the SHA256 output could have null bytes anywhere, so a lot of passwords would evaluate to the same final hash as a lot of other passwords. Which is obviously Not Good. My point is, using raw cryptographic primitives directly can have some serious downsides, and you might want to consider alternatives before doing so.

As a potential alternative, what about using a VPN of some sort such as ZeroTier or Tailscale to connect your clients to the server? Then all you have to do is tell the server to listen only on its VPN address, and all of your traffic will be secured with well-tested crypto at the network layer instead.

If that's not an option (e.g. you don't want users to have to install a separate daemon or something), why not just use Wireguard directly? There are even userspace implementations such as wireguard-go that you can bake directly into your software so that it's completely transparent to the user.

I don't know your specific requirements, of course, so these solutions may not be acceptable. But for what it's worth, although I'm not as religious as some about the Thou Shalt Not Roll Thine Own Crypto thing as some people are, it does strike me as a generally good idea and personally I would only set it aside after making sure I had exhausted all my other options.