I'm trying to explain the full scenario in the next few paragraphs. I think this is important to get some context in which the question is asked, so please bear with me, even if it is a wall of text.
I'm currently tasked with anonymizing data in a secure way. The idea is to use
HMAC(<string to anonymize>, key) to anonymize the data in so that it cannot be reversed. For example if you have a customer ID (
CUST299128218) this would be HMAC-ed using
SECRET as the key to
543a36dd07fe4a3fa4a2db202546eaaccaef71f871ebafe11de3b54784ba266e. Since we want to run analysis over the anonymized data it is important that the same customer ID always yields the same HMAC digest. So we cannot throw away the secret key as we need to anonymize future data with the same key.
The key obviously needs to be stored somewhere safe so it doesn't get out. Otherwise someone knowing a customer-ID could easily find that specific customer in the anonymized data. For various technical/organizational reasons we cannot use a hardware security module for storing the key. So I had a look at HashiCorp's Vault, which seems to be a nice fit for this as it provides a REST API where you can give it a plaintext and it returns the HMAC of this plaintext using a previously stored key. The key never leaves the Vault which is much better than having the key stored in some configuration property of the anonymization software.
However we're talking about large quantities of datasets to be anonymized (a few hundred thousand up to a few millions per day) and it is foreseeable that calling the Vault API for each dataset (possibly multiple times if multiple items need to be anonymized) will result in a ton of overhead that may overburden the infrastructure we have available for this.
Therefore I had this idea: What if I used some fixed string (e.g.
'customer_id_secret_bootstrap') and let Vault create an HMAC on this using the secret key. Then I use this HMAC as secret key for the actual HMAC on the data to anonymize. In functional terms:
temp_key = CALL_VAULT('customer_id_secret_bootstrap') anonymized_text = HMAC( <plaintext>, temp_key)
This way I could only do one call to Vault and keep the temporary key in memory. I should always get the same temporary key back from Vault (since it's an HMAC), but the original key (which is used to derive the temporary key) never leaves the vault and when the program exits, the temporary key cannot be re-created without accessing the Vault. So this way I would ensure the security of the key while not having a million calls to the Vault.
Now knowing that I'm not a security expert by far, this may be a terrible idea for reasons unbeknownst to me. Therefore I'd like to run this with you experts here - can you tell me if this is a good or bad idea and if it is a bad idea could you suggest some alternative approach that would ensure the security of the key and be scalable?
As many answers point out it is not sufficient to just replace the IDs as there are other fields which can be used to correlate information down to a small set of persons or even a single person (e.g timestamps are great for this). We also take care of this by removing or replacing such information to ensure that this cannot happen (we have a very long checklist regarding such things which is based on anonymization standards). I just didn't want to bring in those details here as the question is already very long-winded.