Take the 2-minute tour ×
Information Security Stack Exchange is a question and answer site for Information security professionals. It's 100% free, no registration required.

I have to perform ip address connection stats anonymously because the data posses security risk, so how do I do it?

Is it just like this, that the ip numbers and networks are replaced by random codes, and when operator is sorting out the stats, he can see the codes only?

However, the backend software has the ip numbers there, which are used to update the stats. Should I encrypt this data and generate the codes at once?

Or, I can just compress the data the way, that I group the numbers based e.g. on their country, so e.g. I would have ip number code belonging to UK "hashedipaddr" => "GB", and for this random group, multiple ip numbers would be assigned.

share|improve this question
    
Why should IP addresses be kept anonymous? –  Terry Chia Aug 4 '12 at 0:50
2  
@TerryChia There's many reasons for this. A company may need to send logs to a third-party vendor, but they don't want to give the vendor any specifics about their network's IP space or what particular servers or workstations are at which addresses. Alternately, it may be a matter of anonymizing end-user data to protect the privacy of your users while still being able to gather useful statistics. –  Iszi Aug 4 '12 at 0:53
    
@Iszi I see. Didn't think of that. Thanks. –  Terry Chia Aug 4 '12 at 0:55
1  
Why are you even keeping the IP addresses? IOW, what kind of stats do you need? –  curiousguy Aug 4 '12 at 18:10
3  
ALTER TABLE database DROP COLUMN ip; Even supposedly anonymized data very often isn't. If you don't absolutely need the data, don't keep it at all. –  Michael Hampton Aug 5 '12 at 2:48

2 Answers 2

up vote 7 down vote accepted

If the stats that you are collecting are just as simple as "Have i seen this IP before?" then a Bloom Filter is ideal. Bloom Filter lookups and inserts are both O(1). But most importantly you cannot reverse a bloom filter, not even using brute force due to an unavoidable false-positive rate. You could have an array of bloom filters to put the ip address into categories, such as putting all US geo-ip lookups in the same bloom filter.

A cryptographic hash function or asymmetric cryptography would be less than ideal because its easy to brute force such a small keyspace.

share|improve this answer
    
Many thanks it works the way it's described. Since I am using this data for probability the way I check if set of ips is in the group, this is working all fine for me. –  Andrew Smith Aug 11 '12 at 17:56

This is a quite tricky subject. You don't provide enough information about what you're trying to do for me to provide you a detailed solution, so I'm going to have to stick to general principles:

  • Anonymization is hard and imperfect. While you can try to obscure some of the data (e.g., IP addresses), please understand that this is a very tricky subject. Many attempts at anonymization have turned out to be flawed. Therefore, while it is worthwhile to try to use technical methods to anonymize the data as much as possible, please be prepared for the possibility that they may be flawed.

  • Secure legal and policy approval. Get approval from appropriate policy-level folks at your organization for releasing the anonymized data set. Also, request and obtain a written, signed agreement from the recipient that they will use the data only for certain purposes specified in advance; that they will not share the data with others; that they will use reasonable methods to secure access to the data; that they will report any security breaches to you; and that they will destroy the data upon your request.

  • Apply technical methods. Use technical methods to obscure the IP addresses and anything else that may identify users.

    • IP addresses. The best method to anonymize IP addresses depends in intricate detail upon the intended use of the data-set, and what kind of analysis you want the recipient to be able to do. The most secure method is to delete all IP addresses. If this removes too much information for your situation, you need to tell us more about your situation.

      Other methods that have been proposed include: replacing each IP address with its SHA1 hash (this is not secure; with only 232 possible IP addresses, it is trivial to reverse the hashes and recover the original IP addresses -- so do not use this method); replacing each IP address with a SHA1 hash of the IP address and some 128-bit cryptographic secret (this is much better, though it still allows linking all of the flows with the same IP address, so if you reveal any packet contents, this may identify users and reveal all of their activity); hashing just the first 16 bits of the IP address, again with a crypto key (a bit better still, but still may compromise the privacy of users if you include packet contents).

    • Other packet data. I do not recommend including any payload contents in the data set. Headers are one thing; the payload data is much higher risk.

  • Learn from others. I suggest reading the material in the public research literature on this subject. Here is one good paper:

    There are a number of other research papers on the subject; search Google Scholar for papers that cite this one to find more.

    I recommend that you read this prior research, as it has important lessons for you. Learn from others' mistakes, and others' successes -- it is a lot safer than making those mistakes yourself.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.