How to make database of IP addresses anonymous?

Question

I have to perform ip address connection stats anonymously because the data posses security risk, so how do I do it?

Is it just like this, that the ip numbers and networks are replaced by random codes, and when operator is sorting out the stats, he can see the codes only?

However, the backend software has the ip numbers there, which are used to update the stats. Should I encrypt this data and generate the codes at once?

Or, I can just compress the data the way, that I group the numbers based e.g. on their country, so e.g. I would have ip number code belonging to UK "hashedipaddr" => "GB", and for this random group, multiple ip numbers would be assigned.

Answer 1

If the stats that you are collecting are just as simple as "Have i seen this IP before?" then a Bloom Filter is ideal. Bloom Filter lookups and inserts are both O(1). But most importantly you cannot reverse a bloom filter, not even using brute force due to an unavoidable false-positive rate. You could have an array of bloom filters to put the ip address into categories, such as putting all US geo-ip lookups in the same bloom filter.

A cryptographic hash function or asymmetric cryptography would be less than ideal because its easy to brute force such a small keyspace.

Answer 2

This is a quite tricky subject. You don't provide enough information about what you're trying to do for me to provide you a detailed solution, so I'm going to have to stick to general principles:

• Anonymization is hard and imperfect. While you can try to obscure some of the data (e.g., IP addresses), please understand that this is a very tricky subject. Many attempts at anonymization have turned out to be flawed. Therefore, while it is worthwhile to try to use technical methods to anonymize the data as much as possible, please be prepared for the possibility that they may be flawed.

• Secure legal and policy approval. Get approval from appropriate policy-level folks at your organization for releasing the anonymized data set. Also, request and obtain a written, signed agreement from the recipient that they will use the data only for certain purposes specified in advance; that they will not share the data with others; that they will use reasonable methods to secure access to the data; that they will report any security breaches to you; and that they will destroy the data upon your request.

• Apply technical methods. Use technical methods to obscure the IP addresses and anything else that may identify users.

  • IP addresses. The best method to anonymize IP addresses depends in intricate detail upon the intended use of the data-set, and what kind of analysis you want the recipient to be able to do. The most secure method is to delete all IP addresses. If this removes too much information for your situation, you need to tell us more about your situation.

    Other methods that have been proposed include: replacing each IP address with its SHA1 hash (this is not secure; with only 2³² possible IP addresses, it is trivial to reverse the hashes and recover the original IP addresses -- so do not use this method); replacing each IP address with a SHA1 hash of the IP address and some 128-bit cryptographic secret (this is much better, though it still allows linking all of the flows with the same IP address, so if you reveal any packet contents, this may identify users and reveal all of their activity); hashing just the first 16 bits of the IP address, again with a crypto key (a bit better still, but still may compromise the privacy of users if you include packet contents).

  • Other packet data. I do not recommend including any payload contents in the data set. Headers are one thing; the payload data is much higher risk.

• Learn from others. I suggest reading the material in the public research literature on this subject. Here is one good paper:

  • The Devil and Packet Trace Anonymization, Ruoming Pang, Mark Allman, Vern Paxson, Jason Lee. Computer Communication Review, volume 36 number 1, January 2006.

  • Myths and Fallacies of "Personally Identifiable Information". Arvind Narayanan, Vitaly Shmatikov. CACM, June 2010.

  There are a number of other research papers on the subject; search Google Scholar for papers that cite this one to find more.

  I recommend that you read this prior research, as it has important lessons for you. Learn from others' mistakes, and others' successes -- it is a lot safer than making those mistakes yourself.