Assume I have two servers which are part of a production environment for a web application which handles sensitive user data. The first server is a ruby on rails application, and the second is a postgresql database. The webapp server symmetrically encrypts the data with a private aes key, and then inserts it into a database server.Likewise, when the webapp pulls data from the db it must decrypt it with this key. The webapp frequently needs to pull data out of the database, so all of the data must be decryptable on demand. Currently this key is cached in memory. The webapp connects to the database over a secure VPN tunnel, the database is never exposed to the internet and it is also situated in a different physical location (secure datacenter).

Assume that both servers are running hardened kernels, behind proper firewalls, up to date, utilizing encrypted filesystems, memory address randomization, multi-factor authentication for ssh, etc ... Specifically, I am trying to figure out a good way to protect the private key in a setup such as this. If an attacker compromises the web application server, they would be able to get the private key and then go on to decrypt the entire database.

What I was thinking is that it would make sense to have a third server which is solely responsible for handling all encryption/decryption operations. This server would also be in a different physical location yet on a VPN tunnel which is only accessible to the webapp and database servers. I would have the network setup so that this server could talk to the database and the webapp while the webapp and the database would not be able to communicate with eachother. If at all possible, I would also want to store this key inside of a hardware trusted platform model to mitigate a scenario in which an attacker somehow manages to get into this box.

I believe that this would be more secure than keeping the key on the webapp server because a web application, which must by design be internet accessible, is going to be the most likely attack vector in this security chain. Thus, one really would not want more sensitive information stored there than absolutely necessary. If an attacker managed to break in and get the AES key, it would likely be possible for them to then figure out how to query the database (which this application is permitted to talk to) and do real damage.

At the same time, obviously the same thing is true for any other server where the key is stored that would be handling the encryption. Attacker could just as easily communicate with this server and manipulate it into sending decrypted data back to it.

To mitigate this, I believe that a robust auditing system would need to be in place which could monitor the system across all three servers and flag in real time suspicious queries to the database. For example, a query for a large amount of data, going back further in time than usually necessary for typical operations, or even a large amount of smaller queries for a large amount of individual users data. An engineer would be on call to act if such a report is received at all times. The audit logs would be frequently backed up to another location and inspected at reasonable intervals (like every day).

To summarize, it is my thought process that this setup would mitigate the attack surface considerably because:

  • The encryption server where the master key is stored would not be exposed to the internet
  • The encryption server would be solely used for handling the encryption operations, thus there would be less people logging into it, and only bare minimal software installed on it. Ideally only one or two trusted engineers would ever have access as humans are always the weakest link.
  • It would flag suspicious queries to the database and alert an on call administrator. mitigating the amount of data that could be dumped if it ever was compromised.
  • Theoretically the attacker would have to compromise at least two servers in order to get a large amount of data.

Does this setup sound reasonable? Is there anything critical that I am missing? I would appreciate any insight or advice.

  • I think you basically describe the concept of a HSM, i.e. a dedicated hardware only responsible for cryptographic operations and which is specifically protected against tampering. This is usually done in specifically hardened hardware attached to the device but could also be done using a specifically protected remote device. PKCS#11 provides a standard interface for interaction with such a security module. Jun 18 at 5:35

1 Answer 1


This seems excessively complicated and does not provide significant security improvements, as far as I can see.

You want to protect the confidentiality of the data, and are concerned that the web application might get compromised. To protect a database encryption key from getting compromised, you are thinking of offloading encryption to a separate machine, so that the web application does not have to manage this key directly.

But as you point out, the web application will still be able to request any plaintext data, and it will be decrypted on demand. If an attacker compromises the web application, then the web application and the attacker are indistinguishable. If the web application can request any plaintext data, so can the attacker. This is completely independent of your key management strategy, and in fact completely independent of the question whether you use such field-level encryption at all. The attacker needs the application's access, not necessarily the key.

You are arguing that the attacker might exhibit different behaviour than the web application, and that anomaly detection might help to alert administrator of such deviations from normal patterns. I think that is not a good solution for the following reasons:

  • useful anomaly detection is more difficult than one might think
    • you will be flooded with false positive alerts
  • it will likely only find the most brazen/amateurish attackers, but will be powerless against targeted attacks that only leak individual records or use other evasion techniques
    • when you finally detect an attack, it will likely be after a significant amount of data has already been leaked
  • if the security of your encryption scheme is derived from reliable anomaly detection, you might just as well use this anomaly detection without the complicated key management scheme

How all of this can be resolved depends a lot on the exact business requirements. In some cases, it might be possible to check the user's rights to access the data, instead of the application's right to access the data (compare the confused deputy problem). But with web applications, it is not necessarily reasonable to distinguish users from the web app, since a compromised web app could mislead users to unwittingly provide access, or otherwise exfiltrate or forge credentials.

My guess is that for your scenario, complicated key management schemes will not be part of a solution. But some aspects can definitely help to harden the system. Managing keys with hardware security modules? Great idea! Using encrypted connections (such as VPNs) between servers? Fantastic! Decomposing the system into orthogonal services that are easy to monitor and audit? Go nuts! Intrusion/anomaly detection? All part of a healthy breakfast security posture. Even imperfect security measures do have value, as long as you remember that they merely reduce the likelihood of successful attacks, but that they will not be reliable against a motivated attacker.

Also food for thought: at first approximation, every organization gets compromised eventually. But what happens afterwards? As important as defenses are, disaster recovery is at least as important. Having a server with “less people logging into it” still suggests very manual and stateful procedures for deployment, which will probably inhibit recovery.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.