Mutual TLS replacement

Question

When last time I was thinking about mTLS, I came to a conclusion that this is too hard to implement but at the same time it provides high security. The reason why it's hard to implement is that it's not done on application layer, so e.g we can't require mTLS for some endpoints, but we need to require mTLS for whole domain. What's more, we need to have certificate rotation mechanism for any app that uses mTLS (and if it mechanism would break, our services will be unavailable until we rotate certificate). So I thought that for my own needs I could design my own protocol (on application layer) with a similar level of security that mTLS provides.

Let's see what wikipedia says about what kinds of attacks mutual authentication prevents (https://en.wikipedia.org/wiki/Mutual_authentication#Defenses):

1. Man-in-the-middle attack
2. Replay attack
3. Spoofing attack
4. Impersonation attacks

So consider what we can do to prevent these attack without mTLS.

1. Preventing MITM: client sends signed request with e.g. pre-registered ed25519 to some server. This server must verify signature and reject it, if signature isn't valid. If signature is valid, we can say that we have authenticated client and his request wasn't modified (cause of it was signed). Server response must also be signed.
2. Preventing Replay attack: we signed our request earlier, so we can easily add expiration date to this signed request. Now, older request and responses (because response also must be signed) cannot be used. What's more, we can create e.g. 32 cryptographically strong random bytes (nonce) every request and we would to add it in response also. Then client can verify received nonce with this created when sending request.
3. Preventing Spoofing attack: I believe it's done by signing requests and responses. (please correct me if I'm wrong)
4. Preventing Impersonation attacks: I think it's also done by signing, because only the client who has signed the request know private key. If private key leaks, this is as dangerous as SSL certificate private key leak, so I think we shouldn't carry about it.

Let's consider another option if we don't trust that client is setting a really short lifetime for his requests: we can use TOTP.

I'm not an security expert, so I'm asking because I could assume some things wrong. Please rate security of above solution (may be compared to mTLS) and tell me if I missed some possible attacks that mTLS prevents.

I should also add that I think about it in context of OAuth2/Oidc. It means all client's are pre-registered (their public keys to verify signature also). Client can verify server response by getting public key from /.well-known/openid-configuration endpoint.

NOTE: I know you can say that implementing mTLS seems to be simpler to implement than my idea, but I need to have something like mTLS (high security) only on some endpoints and I can't create subdomains or domains intended for mTLS. Also note that delayed certificates (renegotation) is explicitly banned in HTTP2 so it's not solution for my problem.

Answer 1

Here are some questions to think about:

1. "client sends signed request with e.g. pre-registered ed25519 to some server" - If you don't use TLS, how will client know that it talks to the server and to a man-in-the-middle?

2. How will your server know who is trying to pre-register? How will server distinguish real clients from the man-in-the-middle?

3. "Server response must also be signed." - How will clients know, if the signature belongs to your server and not to the man-in-the-middle? In case of TLS it is trivial. How are you going to address that?

4. If your server was compromised (configuration error, social engineering, bug in software, etc.), how will clients know that the old signature should not be trusted any more? In case of TLS it is trivial. How are you going to address that?

5. You don't write any word about encryption. Does it mean you are not going to encrypt the traffic? If you want to encrypt it, how are you going to share the encryption key?

6. How about Perfect Forward Secrecy? It seems you are not going to implement it. In particular this means that if an attacker obtains server keys, it will be possible to decrypt any previous traffic that was intercepted. Where as in case of TLS there is no way to decrypt any previously captured traffic, even if you know server keys.

Answer 2

There are many details missing from your question, so my answer assumes a lot. But here we go:

• how many types of clients will be able to use your custom protocol? Unless you are able to provide the appropriate libraries for others to use, nobody is going to use it
• you seem to have already taken for granted a trusted third party; TLS requires CAs, you require a public key registry (for the e.g. pre-registered ed25519, you mention). If a certificate rotation mechanism is a problem for you (as you state in your question) then the registry should not match your criteria either
• TLS provides confidentiality and forward secrecy. Your solution does not; if the data exchanged contain confidential information (like passwords) then your protocol is not suitable to use
• there's a reason why application layer security is not usually used as you describe it or intent to; it costs a lot, in terms of CPU and RAM resources. For example, if two REST endpoints require TLS, then each endpoint would need to do exactly the same thing (authentication, key agreement, en-/decryption of data etc) in order to secure their communications. This means that now you have doubled the load on the server. For more REST endpoints you'll require a lot more resources; this is not cost effective

Also some thoughts on the attacks you mention:

Preventing Replay attack: we signed our request earlier, so we can easily add expiration date to this signed request. Now, older request and responses (because response also must be signed) cannot be used. [...]

Expiration date means, in principle, a window of opportunity for a replay attack. Even if you use network time synchronization in order to allow for very short expiration periods, by performing all the cryptographic operations at the application level you've introduced variability in the time that it takes to process a request (depending also on the number of clients hitting several endpoints) so the expiration can't be too soon

[...] What's more, we can create e.g. 32 cryptographically strong random bytes (nonce) every request and we would to add it in response also. Then client can verify received nonce with this created when sending request.

If this is to be used without the expiration concept, then it's not ideal; now every peer needs to remember all nonces that it has seen so as to reject the replayed ones

Preventing Spoofing attack: I believe it's done by signing requests and responses.

Preventing Impersonation attacks: I think it's also done by signing, because only the client who has signed the request know private key. If private key leaks, this is as dangerous as SSL certificate private key leak, so I think we shouldn't carry about it.

This depends on whether the key used to sign the data is somehow connected with the id of the sender (e.g. name or IP on its corresponding public key). If that's the case, then you're using a lot of the functionality that you said you don't want to.

Finally, a solution to your problem would be to have the protected endpoints accept connections to one port and the non protected to another port. Then have the server that serves the protected endpoints use mTLS and the other not. That would be a lot easier in my opinion.