How to design a single-request, non-CA-based custom server-to-server authentication method?

Question

We have a series of services exposed as web servers. Potentially, every web app could contact and be contacted by any other one. I want to secure the communications between the applications, which means ensuring that both parties are who they say they are. I want my applications to only be allowed to communicate between themselves: this is not a matter of frontend-backend communication, and no app is in the hands of customers or external users.

That said, I also want to avoid the complexity of setting up a CA, generating TLS certificates and so on. We want the security checks to be as streamlined as possible, and possibly easy-maintenance and easy to migrate. Also, we are going to add new applications, and the current method (which I'll describe in a second) is already too cumbersome according to my manager.

I want to understand what the risks are in a scenario in which we implement a quasi-PKI infrastructure, which means I would use industry-recognized encryption algorithms (no custom encryption at any stage) but in a custom workflow.

Let's consider a single connection between a provider and a consumer of the service, and for simplicity's sake let's call them server and client.

All connections are HTTPS-based, except maybe when developing. This should take care of MitM attacks and of server authentication.

What we currently have is as follows. The server has a registry of authorized client applications, and each of them has a GUID and a secret key. When a client contacts the server, it sends its GUID in a custom HTTP header; the server recognizes the GUID from the registry and challenges the client to an authentication challenge (essentially HMAC): a request is sent to the URL stored in the registry, containing a random string, the client responds with the result of encrypting the random string using its secret key (for example with RSA-256). If the server receives the expected response, then the client is authenticated and the response is served.

The secret key is never sent over the network, which keeps it safe, and the challenge string changes every time, which protects against replay attacks.

The weakness is that, if a third party learnt the client's application GUID, they could make requests to the server and the server would send the challenge to the real application, without noticing. This can be fixed either by checking the provenience of the request (but I think that can be spoofed) or by making the client record its requests, so that it rejects authentication challenges that do not match a request it made.

However, this system requires us to define the client application in the registry, pair the access information in the registry and in the client's configuration, and implement the challenge handler in the client. Also, access information may be different between development, staging and production (I'd say it's best it be different). If we factor in 15-20 applications (and growing), each managing its registry as a server, each potentially being a client of several others, you can see this becomes an important setup overhead.

What I'm aiming for is to remove the authentication challenge. Removing the authentication challenge speeds up the process and eliminates the need for a challenge handler. A server would still keep a registry of authorized clients, with a secret key for each client.

Instead of HMAC, the challenge would be time-based, even though I don't like having an interval during which the result would stay the same (which exposes to replay attacks for a short time). However, I can't figure another way to have an always-changing authentication code without the server offering a seed.

So, to sum up, the question is two-fold:

• is it any less secure to use industry-standard encryption algorithms in a custom workflow (assuming the custom workflow doesn't introduce security holes itself)?
• how to perform the authentication without the authentication challenge, without also exposing to replay attacks?

Answer 1

Here's a few thoughts and suggestions.

(1)

All connections are HTTPS-based, except maybe when developing. This should take care of MitM attacks and of server authentication.

At this point you need to have a PKI in place to ensure integrity of connections;

(2)

What we currently have is as follows. (...) The secret key is never sent over the network, which keeps it safe, and the challenge string changes every time, which protects against replay attacks.

Your methodology doesn't look bad at face value, however the major issue is key distribution. How do you handle/store/distribute these secret keys?

(3)

The weakness is that, if a third party learnt the client's application GUID, they could make requests to the server and the server would send the challenge to the real application, without noticing. This can be fixed either by checking the provenience of the request (but I think that can be spoofed) or by making the client record its requests, so that it rejects authentication challenges that do not match a request it made.

I'm not sure how you can verify the provenience of this value. Also sending the challenge code to a spoofed client shouldn't be a security issue in your implementation. As in, this should not be of use for an attacker, as it would if it were netntlm.

(4)

What I'm aiming for is to remove the authentication challenge. Removing the authentication challenge speeds up the process and eliminates the need for a challenge handler. A server would still keep a registry of authorized clients, with a secret key for each client.

Sounds like adding a weakness and complexity.

is it any less secure to use industry-standard encryption algorithms in a custom workflow (assuming the custom workflow doesn't introduce security holes itself)?

You are asking a question with a caveat where the caveat is the answer. If you design and implement a secure system, then the system will be secure. The matter is, however, can you be confident of this?

how to perform the authentication without the authentication challenge, without also exposing to replay attacks?

This is a long winded response. A secure authentication protocol needs resilience against brute forcing, replay attacks and in some ways impersonation of authenticated peers. It is up to you to work out how to do this.

I understand this is the answer you want to avoid. But using PKI with client side certificates addresses most of these points, certainly (1) (2) and (3), and you might be able to forego (4).