Suppose I want a server to provide some simple, primarily file related functionality and I want to allow access to all users that have a certain public key (server has the private key) and password to the server. It would be a server that few people would know the purpose for (IP would be explicitly shared only with a handful of people), and all its ports besides the one for this functionality would be non-listening. None of the data is important, the main goal is to just avoid any potential issue with someone attacking the server for the heck of it.

What would be the issues/weaknesses with the following approach? I'm not really interested whether it's inefficient, just what the security risks are if I can assume an attacker does not have access to the public key, private key, or the password.

  1. Client encrypts password with public key and sends it over to the server. Server decrypts it, validates the password, or TCP session ends. The password is distributed to users, and not possible for users to assign, so hashing it does not seem crucial.
  2. Client generates a symmetric key (AES), encrypts it with public key and sends it over to server.
  3. Server creates a nonce, encrypts it with symmetric key, and sends it to client. Client decrypts it, concatenates it with a string T (hardcoded constant known to clients and server), encrypts it, and sends to server. Server validates that it received nonce + T, else session ends.
  4. Data transfer starts, using the established symmetric key.

I can see there's no authentication of the server - but I can't really any issue with that.

  • 2
    What you are describing seems like a hacked-up version of TLS. Why not just use TLS, which is tried and true, and has been proven, tested, and vetted, instead of breaking the second commandment of information security (don't roll your own secure protocols). In addition, you'll have server authentication (by way of the server certificate), and client authentication can be done by way of a client certificate, or a password sent in the request from the client to the server.
    – mti2935
    Jul 1, 2022 at 20:34
  • Is there anything preventing an attacker from snooping the encrypted password (in step 1), and just reusing it to make their own connections to the server (similar to a pass-the-hash attack)? Jul 1, 2022 at 21:29
  • @mti2935 My reason for using this is that I haven't actually written the code, but I would like to use the app. I could rewrite it to use TLS, but I'm a bit afraid, and don't want to touch code that seems to work - so right now I'm first trying to get an understanding of what it does, and whether it's worth it for me to modify it myself.
    – Jake1234
    Jul 1, 2022 at 22:05
  • @GordonDavisson I'm not sure what the term snooping means, sorry. Yes, someone could establish their own connection using a replay attack, but they will not get past 3 due to not being able to decrypt the nonce. This is actually the only modification I'm willing to do to the code (adding a nonce), because adding it after an established, encrypted connection seems easy enough.
    – Jake1234
    Jul 1, 2022 at 22:11

2 Answers 2

  1. Sure.
  2. Okay. The acceptance of both these points relies on me choking on the gut reaction to write "We don't encrypt with asymmetric keys directly."
  3. I don't really understand this one. The client is the one that sent the key to you, not sure why it's necessary to do this challenge and response. If T is some sort of secret, how does it differentiate from the password in step 1?
  4. Fine


This has ZERO forward secrecy. Compromise of the server private key will enable decryption of all communications, even at a later date. Compromise of the symmetric key by an active attacker in step 2 will lead to decryption of step 4.

As mentioned, writing your own secure protocols is kinda a big no-no. If TLS is a bit much for you, I'd suggest looking at the Noise protocol. The NKPsk0 pattern seems to be what you want (server is known by their asymmetric, clients are known by their preshared secret).

  • The 3. is done to avoid replay attacks. Without some sort of nonce, it seems to me like there is nothing stopping a complete replay.
    – Jake1234
    Jul 1, 2022 at 22:01
  • T could be the password, but I don't think there's any need for that - we just need the modified nonce to be different than the original one sent by the server. I'd assume that modifying the input of AES even slightly (i.e. some bit shifting, or concatenation with some fixed string) is very likely to give a completely different output.
    – Jake1234
    Jul 1, 2022 at 22:21

There are two potential downsides that strike me here:

  1. Complexity. This is a fairly complex arrangement, I am assuming that you control the software that does the actual crypto on the client side but even so, in my experience this sort of thing tends to break all the time, and eventually your users get fed up with you and go find some other solution. (unless the user is just you, in which case you just end up spending way more time than necessary fixing it.)

  2. You're rolling your own crypto here, to a certain extent, which is generally discouraged because it's very easy to make subtle mistakes that don't break anything (in the sense of making it unusable), but make your whole arrangement useless. For example, I don't recall the precise details but a while back there was a story about a widely used implementation of some hashing algorithm (possibly bcrypt?) that assumed a null byte indicated the end of a string, so if you passed it a sequence with a null byte it would disregard everything following. Eventually, someone decided to pass it the output of a different hashing algorithm, I think SHA256, so that it would be possible to use arbitrary-length inputs. Only problem is, the SHA256 output could have null bytes anywhere, so a lot of passwords would evaluate to the same final hash as a lot of other passwords. Which is obviously Not Good. My point is, using raw cryptographic primitives directly can have some serious downsides, and you might want to consider alternatives before doing so.

As a potential alternative, what about using a VPN of some sort such as ZeroTier or Tailscale to connect your clients to the server? Then all you have to do is tell the server to listen only on its VPN address, and all of your traffic will be secured with well-tested crypto at the network layer instead.

If that's not an option (e.g. you don't want users to have to install a separate daemon or something), why not just use Wireguard directly? There are even userspace implementations such as wireguard-go that you can bake directly into your software so that it's completely transparent to the user.

I don't know your specific requirements, of course, so these solutions may not be acceptable. But for what it's worth, although I'm not as religious as some about the Thou Shalt Not Roll Thine Own Crypto thing as some people are, it does strike me as a generally good idea and personally I would only set it aside after making sure I had exhausted all my other options.

  • I would probably not use the approach used in the code I'm working with, because I'm simply inexperienced and I don't know what I'm doing. But I'm stuck with it, and I'm not sure whether it's worth rewriting given my inexperience, and I don't want to bother the dev with it either - it really is a very non-public app (bug tracking app, clients can post bug reports), and I don't want to bother the dev about rewriting it using TLS.
    – Jake1234
    Jul 1, 2022 at 22:31

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .