When last time I was thinking about mTLS, I came to a conclusion that this is too hard to implement but at the same time it provides high security. The reason why it's hard to implement is that it's not done on application layer, so e.g we can't require mTLS for some endpoints, but we need to require mTLS for whole domain. What's more, we need to have certificate rotation mechanism for any app that uses mTLS (and if it mechanism would break, our services will be unavailable until we rotate certificate). So I thought that for my own needs I could design my own protocol (on application layer) with a similar level of security that mTLS provides.
Let's see what wikipedia says about what kinds of attacks mutual authentication prevents (https://en.wikipedia.org/wiki/Mutual_authentication#Defenses):
- Man-in-the-middle attack
- Replay attack
- Spoofing attack
- Impersonation attacks
So consider what we can do to prevent these attack without mTLS.
- Preventing MITM: client sends signed request with e.g. pre-registered ed25519 to some server. This server must verify signature and reject it, if signature isn't valid. If signature is valid, we can say that we have authenticated client and his request wasn't modified (cause of it was signed). Server response must also be signed.
- Preventing Replay attack: we signed our request earlier, so we can easily add expiration date to this signed request. Now, older request and responses (because response also must be signed) cannot be used. What's more, we can create e.g. 32 cryptographically strong random bytes (nonce) every request and we would to add it in response also. Then client can verify received nonce with this created when sending request.
- Preventing Spoofing attack: I believe it's done by signing requests and responses. (please correct me if I'm wrong)
- Preventing Impersonation attacks: I think it's also done by signing, because only the client who has signed the request know private key. If private key leaks, this is as dangerous as SSL certificate private key leak, so I think we shouldn't carry about it.
Let's consider another option if we don't trust that client is setting a really short lifetime for his requests: we can use TOTP.
I'm not an security expert, so I'm asking because I could assume some things wrong. Please rate security of above solution (may be compared to mTLS) and tell me if I missed some possible attacks that mTLS prevents.
I should also add that I think about it in context of OAuth2/Oidc. It means all client's are pre-registered (their public keys to verify signature also). Client can verify server response by getting public key from
NOTE: I know you can say that implementing mTLS seems to be simpler to implement than my idea, but I need to have something like mTLS (high security) only on some endpoints and I can't create subdomains or domains intended for mTLS. Also note that delayed certificates (renegotation) is explicitly banned in HTTP2 so it's not solution for my problem.