Should I send (en_dek + data) to the key server or should I fetch the KEK into the app server? Which one is more safe practice?
Sending the key en_dek to the key server to be handled there is pretty much how a hardware security module works at a high level. The advantage of this approach is that the key material never touches the application server. So any compromise of the application server can only decrypt the keys that are currently in memory, as well as potentially submit decryption requests to the key server. However, the attacker does not automatically get access to the master key itself, which is important.
Given you're already talking about sending the data to another system to be decrypted, I wonder if it might not be better simply to send the data, thereby hiding the key entirely from the application server?
In any scheme like this, the security of the key depends entirely on the security of the server or HSM on which it is stored. Usually, true HSMs are reasonably good at protecting keys unless you're very handy with electronics.
Both these methods have a flaw that if an attacker is able to get access to either of these servers, he will get to know both KEK and DEK. How can we prevent this?
You can't, except by using a dedicated HSM, which is essentially what you're discussing building. I would simplify the scheme and keep the keys on one server dedicated only to cryptographic operations, and keep that server locked down.
For more on these two points, see the OWASP rules on cryptographic storage.
When we talk about key rotation, which of the two keys (DEK and KEK) is rotated?
Ideally, both. Rotating the KEK should be easier, since DEKs are fixed size and there shouldn't be all that many of them. Rotating DEKs might be harder if you are encrypting lots of data.
From the OWASP page, which says it better than I can:
Rekeying refers to the process of decrypting data and then re-encrypting it with a new key. Periodically rekeying data helps protect it from undetected compromises of older keys. The appropriate rekeying period depends on the security of the keys. Data protected by keys secured in dedicated hardware security modules might only need rekeying every three years. Data protected by keys that are split and stored on two application servers might need rekeying every year.
