I'm wanting to share datasets between two organisations in a pseudonymised way such that they link together via mutual entities (e.g. people) but so that the entities cannot be identified. E.g.:

Organisation A has dataset A. Organisation B has dataset B. Person C appears in both datasets, identified by their unique ID 12345. I want to store both datasets together (in a data warehouse outside of both organisations), linking activity for person C in dataset A with activity for person C in dataset B. However, to ensure person C cannot be identified, I want to store their unique ID (across both datasets) in a hashed way - such that it cannot be reversed (e.g. such that 12345 becomes 29CX9).

Option 1

I could simply hash the unique identifier via SHA2 512 (or other algorithm) and store this in the warehouse therefore providing a pseudonymn that always maps to the same output for a given input. However, unlike with passwords where one-way hashing is effective, anyone (that knows the hashing algorithm) could run a known unique identifier through the hashing algorithm and get the hashed pseudonym. This method prevents people reversing the hash but as identifiers aren't secrets in the same way passwords are, it doesn't really provide any security.

Option 2

Create a table that assigns a new randomly generated unique identifier for each identifier we want to create a pseudonym for and then lock this table down so no one has access. This gives us a unique identifier that isn't related to the source identifier so someone who knows a source identifier can't find out what the pseudonym would be in the shared warehouse, however it does then mean that if the table is ever breached, an attacker has everything they need to identify all entities in the warehouse.

Further Options...?

Are there further options? Does it essentially boil down to one of these two options - either we can protect a source identifier from being converted to its pseudonym or we can protect a pseudonym from being reversed to its source identifier but not both? Would the best bet be a hybrid were the source identifier is hashed (SHA2 512) and then a locked down lookup table is created that maps hashed identifiers with unique randomly generated identifiers...?

  • 1
    There may be a good reason why this is a bad idea, but what about using an HMAC? Keep the key safe and no one will be able to reproduce your identifiers. – Jay Oct 22 '15 at 18:47
  • 1. Why do you want to link that data? 2. Will you need to add more data to person C at a later date? – Gudradain Oct 22 '15 at 20:08
  • If the IDs are [all exactly 128 bits] or [all less than 128 bits] then you could use AES with no mode of operation or bit padding followed by that. ​ (This would avoid the cost of storing a potentially huge table.) ​ ​ ​ – user49075 Oct 22 '15 at 21:39
  • @Jay I could use a HMAC although in this instance I'm not sure it would be much different than just using a static salt/key as we don't need to verify the message. It does mean that people can't simply reproduce the hash by running the source data through the same algorithm but (and I may be going the nth degree here) if someone managed to obtain/crack the key, they'd then have access to as much data as they wanted (given access to source identifiers) – Drunk Goldfish Oct 23 '15 at 11:05
  • @Gudradain 1. For use in high level, BI analytics - we're not interested in the entities themselves - more interested in trends in the data. 2. Yes we would, we'd be doing iterative loads that could add new data against person C each load – Drunk Goldfish Oct 23 '15 at 11:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.