Please keep in mind that without hardware security (TEE, Some crypto key ladder, etc...) a real strong security mechanism can't be implemented as long as potential attacker has physical access to the server.
Now, to implement the "good" security mechanism for your need, you have to list every possible attacks you must be resilient to:
Below is an example of a security mechanism that is supposed to protect against replay attack.
Initial setup:
- AppA and AppB both have a pair of public/private key for asymetric crypto.
- AppA and AppB both knows other app public key.
Security mechanism:
- AppA generate a payload that is composed of:
- A magic constant: let's say "PLOP".
- A random token: nonceA.
- A timestamp: timestampA.
- AppA cipher this payload with AppB public key.
- Using chosen communication mechanism, AppA push encrypted payload to AppB.
- AppB unciphered payload and checks if:
- magic constant is at very beginning of the clear payload.
- timestamp is almost equal to now().
- If previous checks were ok, than AppB knows that AppA is legitimate, it should now proves it is too, so...
- AppB generate a payload composed of:
- Same magic constant PLOP
- Previous random token nonceA
- A newly generated random token nonceB
- A timestamp timestampB
- AppB ciphers payload with AppA public key and push result to AppA.
- AppA just do the same operations done by AppB:
- Check magic constant is present.
- Check if provided NonceA is the same.
- Check timestamps consistency.
- Then AppA must return to AppB a payload with NonceB, timestampB and ciphered with AppB public key.
- At this point, both apps ensured that other one is legitimate.
This is one of possible mechanism to fullfil your requirements, there's probably some way to improve this scheme using cryptographic signing, etc...
Nevertheless, keep in mind that:
- With a total control over the physical machine, there's no way to 100% fullfill your requirements with no secure chipset because:
- Crypto keys can be extracted from binaries or memory.
- Certificate Authority can be modified and then, SSL protections are not sure anymore...
- To protect against replay attack, use timestamp and nonce.