How can I authenticate a client connection in a secure way?

Question

Let's say I have a server listening on port 1234. I have some client software that needs to be able to connect to this port. But I want to prevent malicious users from bypassing the client software and connecting to the port by some other means (such as in a console or another piece of software).

The trusted client software and the server may share a secret key, if necessary (I can live with the possibility that the key can be extracted from the binary). I'd prefer not to send such a key in plaintext, but data after the authentication can be in plaintext. Specifically, I'm trying to figure out how to defeat a man-in-the-middle attack where the malicious user is using the trusted client software to calculate correct responses to server challenges.

Can I get there from here?

I could have the server's listening port bind only to localhost and require that clients first gain access to the machine via ssh. Then the client software could use an ssh library to run a command on the server that connects to the local port (in this scenario, the malicious user would be unable to use ssh to access the machine because he would not have the password). But then all my traffic is encrypted, which is additional overhead. Perhaps there is a program similar to ssh that only does the authentication but then leaves the channel in plaintext after that?

Update:

I ran a test to determine the overhead associated with encrypting all traffic.

spew.rb outputs 10 million 100-character lines.

CONTROL:

fantius@machine> time /home/fantius/spew.rb > /dev/null

real    0m35.015s
user    0m34.934s
sys     0m0.084s

top shows 25% cpu usage (one full core, of four cores)

TEST:

fantius@machine> time ssh localhost /home/fantius/spew.rb > /dev/null

real    0m40.704s
user    0m19.981s
sys     0m1.400s

top shows 45% cpu usage (almost two full cores)

That message rate probably exceeds most inter-machine connections.

Answer 1

First, I suggest avoiding assumptions about encryption overhead. You should measure. My PC from 2001 was already powerful enough to transfer data with SSH at the full speed of a 100 Mbit/s link, with encryption. My actual PC, a common Core2 at 2.4 GHz, can do AES encryption at 167 Mbytes per second (yes, megabytes, not megabits), more than enough for gigabit ethernet, and that's using a single core -- I have three other cores.

If you want to authenticate a client then the client must have something special that the envisioned malicious user has not. Cryptographically speaking, some piece of secret data, which we will call a key because that's what keys are: a piece of secret data, used in an algorithm which is assumed not secret. Note that this entails a structural flaw which cannot be really avoided: anybody with read access to the code of the client may perform some reverse engineering and recover the key. This can be made a bit more difficult through code obfuscation techniques, but not to the point of making it impossible.

Also, you want to authenticate the connection, i.e. not only the act of initiating the data tunnel, but also the data which is actually transfered. Otherwise, the attacker could hijack the connection once the authentication step has been performed. This implies using a cryptographic integrity check, which is called a MAC. At that point, adding encryption is easy. SSH uses both symmetric encryption and a MAC.

Out of existing building blocks, I suggest the following solutions:

1. Use SSH. The client connects through SSH and uses a hardcoded password to authenticate itself (the password is then the key I was talking about above). Man-in-the-middle is avoided by the usual mechanism of SSH, i.e. the SSH client should keep a copy of the server public key.

2. Use SSL/TLS. This is very similar to what happens with SSH. The client knows that it talks to the correct server because the server public key is presented as part of a X.509 certificate, which is signed and can be validated by the client. The client must have knowledge of the certification authority public key. Alternatively, the client may already know the server certificate and just check that this is the same certificate than the one sent by the server. Once the tunnel is established, with encryption and integrity checks, the client may just send his key "as is" to prove its identity.

3. Use SSL/TLS with a client certificate. In this setup, the client has a public/private key pair. The private key is the key I was talking about. The public key is encoded into a certificate that the client shows during the SSL handshake. All of this is standard SSL stuff, hence supported by existing SSL libraries. Compared to solution 2, this avoids the need to include a key-sending message in your application protocol, and it also allows the use of MAC-only cipher suites, in case it turns out that encryption is too computationally expensive (which I doubt).

4. Use the client key in a Password Authenticated Key Exchange in order to derive a common session key, then used for a MAC (and possibly encryption). The point of PAKE protocols is that they can tolerate low-entropy secrets, i.e. passwords which fit in the brain of human users. Running a MACed tunnel has a few subtleties so the proper way is to use an existing protocol. This then calls for TLS with SRP. The "password" is hardcoded in the client (and although SRP tolerates short passwords, nothing prevents you from using a long and very random password). The beauty of the thing is that this suppresses any mucking with certificates or known public keys. The client is authenticated by virtue of its knowledge of the password. Simultaneously, the server is also password-authenticated by the client. As an added bonus, with SRP, the server only stores a password-derived piece of data, which is not directly usable by the client, so an attacker reading the server database does not gain automatic access as a "client".

My favorite solution is of course the one with SRP. It is elegant. Unfortunately, SRP is rather new, so not all SSL/TLS implementation support it. But GnuTLS does.

Answer 2

When everything is all said and done, there is no real way to prevent a 3rd party client from connecting to the port if everything is designed to run without human interaction.

The only way you could prevent this problem is if you have a non-hackable (or more accurately, a harder to hack) system that provides the identification piece, i.e. a user, and those credentials are transferred in an encrypted/hash way. If any "secret" piece of information that you plan on using as a credential is stored within the application some how, a malicious user will be able to access that information.