Mutual authentication between application interfaces are mandated in my place of work.

We are looking to implement a vendor provided software where they are installing two "components" on a single Windows Server. One is an API sitting on IIS, the other is an application.

How can this be implemented, here are my thoughts:

  • A application user can be created so the application can authenticate the API but this is one way authentication, not mutual.

  • Is it possible to have two separate SSL certs on one server, one for each component and implement CN/DN checking for Mutual Authentication? If so how can this be implemented.

  • Worst case is we can separate the components onto two servers and implement a TLS connection with CN/DN checking for Authentication. However this is the more costly option.


1 Answer 1


Please keep in mind that without hardware security (TEE, Some crypto key ladder, etc...) a real strong security mechanism can't be implemented as long as potential attacker has physical access to the server.

Now, to implement the "good" security mechanism for your need, you have to list every possible attacks you must be resilient to:

Below is an example of a security mechanism that is supposed to protect against replay attack.

Initial setup:

  • AppA and AppB both have a pair of public/private key for asymetric crypto.
  • AppA and AppB both knows other app public key.

Security mechanism:

  1. AppA generate a payload that is composed of:
    1. A magic constant: let's say "PLOP".
    2. A random token: nonceA.
    3. A timestamp: timestampA.
  2. AppA cipher this payload with AppB public key.
  3. Using chosen communication mechanism, AppA push encrypted payload to AppB.
  4. AppB unciphered payload and checks if:
    1. magic constant is at very beginning of the clear payload.
    2. timestamp is almost equal to now().
  5. If previous checks were ok, than AppB knows that AppA is legitimate, it should now proves it is too, so...
  6. AppB generate a payload composed of:
    1. Same magic constant PLOP
    2. Previous random token nonceA
    3. A newly generated random token nonceB
    4. A timestamp timestampB
  7. AppB ciphers payload with AppA public key and push result to AppA.
  8. AppA just do the same operations done by AppB:
    1. Check magic constant is present.
    2. Check if provided NonceA is the same.
    3. Check timestamps consistency.
  9. Then AppA must return to AppB a payload with NonceB, timestampB and ciphered with AppB public key.
  10. At this point, both apps ensured that other one is legitimate.

This is one of possible mechanism to fullfil your requirements, there's probably some way to improve this scheme using cryptographic signing, etc... Nevertheless, keep in mind that:

  • With a total control over the physical machine, there's no way to 100% fullfill your requirements with no secure chipset because:
    • Crypto keys can be extracted from binaries or memory.
    • Certificate Authority can be modified and then, SSL protections are not sure anymore...
  • To protect against replay attack, use timestamp and nonce.
  • This answer is a good example of some of the steps in a threat model. The original poster is asking about mitigations to vulnerabilities without clearly stating what the threats are that might take advantage of those vulns. List the threats first, and then evaluate potential mitigations. A good book on threat modeling will probably help. Jan 2, 2019 at 19:35
  • "Certificate Authority can be modified" - If you control both the client and the server, your app can use a self-signed certificate and code the app to require that specific certificate. One reason CAs are needed is to solve the key exchange problem (note that my proposal does not resolve the key exchange for the initial distribution of the client and server, but the attack window is much narrower under my proposal, since only the initial distribution/setup is vulnerable to bad CAs, rather than all communication) . Technically, a CA works here, too. The key feature here is pinning.
    – Brian
    Jan 3, 2019 at 14:57

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .