7

On Monday, some IP address xxx.xxx.xxx.xxx visits my site. I log it.

On Tuesday, another IP address yyy.yyy.yyy.yyy visits my site. I log it.

If the two IP addresses are equal, I throw out the second one because I'm only interested in unique visits to my site. If they are not the same, I keep both in my log.


Question: Is it possible to do this without ever logging the IP addresses themselves, by using some kind of hashing/encryption, in such a manner that it is not possible to deduce the IP addresses from the hashed/encrypted data?

At first glance, it seems to me that this would be impossible, because you can always brute force all the possible IP addresses into whatever hash/encryption scheme you're using, until it matches your hashed/encrypted data - and thus deducing the original IP address.

But I hope I'm missing something. Maybe it's possible to bring time into this to make this achievable somehow?

I'm not knowledgeable enough in security to know the possibility or impossibility of this. Any advice would be appreciated.


Additional information: I'd like to store as much information as possible (without, of course, being able to deduce the original IP addresses). For instance, I may not necessarily want to store merely the all-time total of unique visitors. Instead, I may want to know the total unique visitors during a particular day, week, or a month. So actually storing all of the encrypted/hashed IP addresses would be important. So, in this sense, I was wrong when I talked above about "throwing out the second one".


Note: I'm looking for some scheme under which nobody, not even me, can deduce the IP address. Some solutions have been provided that are such that I can myself technically deduce the original IP address, but I don't even want ability to do so myself. This sounds impossible, but who knows. Maybe some trickery exists out there for achieving what I want?

4
  • Yes, if one wanted to he could do so, however please state the applications of this system so that a better answer can be given. Do you merely want to know the total no. of unique visitors, or do you want additional tracking for which storing the hash of each unique visitor's IP is critical?
    – Rohan
    Aug 11, 2012 at 5:38
  • @RohanDurve-Decode141 Thanks, I have now updated my question accordingly. I want as much tracking as possible, not only the all-time total. Aug 11, 2012 at 5:53
  • What's wrong with md5hashing the IP addresses? Also, it seems like IP addresses are all over the Internet, and you're not associating the IP address with user data. Sorry, I'm not sure I understand the sensitivity of protecting an IP address. Maybe you could explain? Thank you! :)
    – jmort253
    Aug 11, 2012 at 5:56
  • @jmort253 If I only MD5 hash it, I can always deduce the original IP address by just feeding all the possible IP addresses through the MD5 hash until the result matches the hash I have stored. It's important to me to be unable to track the IP address because I want maximum privacy for my individual visitors - but still being able to have some general data about unique visits, countries, etc. - but never individual IP addresses. Aug 11, 2012 at 6:01

6 Answers 6

8

First of all: Why are you so concerned with not storing IP addresses? It's not private data; the OS or NAT router happily includes it with every sent packet. Okay it's kind of necessary, but what I mean is that they aren't really private. Also, random IPs are not really useful for hackers, and it's not a privacy concern when you can only see them yourself.

Secondly, it's not really useful to save IP addresses with the goal of counting unique visits. Not only can one IP have hundreds or thousands of users (like schools), but a single user can also easily occupy a /16 subnet or two (many people's IP changes every few days or so), not to mention laptops and mobile phones. A better technique would be session cookies, which are unfortunately restricted to the browser (and thus one device, so not counting people but the number of browsers really), but it's a lot better than IP addresses.

Lastly, if you still want to go with storing IP addresses, there are basically two ways:

  1. Bloom filter, as mentioned before. This is not 100% correct though, but probably good enough for this purpose.
  2. Using a hashing function which takes forever to compute. Sha-2'ing the IP a couple million times, it's impractical to brute-force it.
    • (There are about 3.7 billion globally usable and unique IP addresses, so if it takes 50ms to compute one hash, it takes 2.9 years before you have 50% chance of brute-forcing one of them, not taking Moore's law into account.)
4
  • This is sort of what I was getting at in my comment. IP addresses != user data.
    – jmort253
    Aug 11, 2012 at 22:31
  • 2
    Thanks for this reply. The reason I don't want to store my users' IPs is to guard against the extremely unlikely case where a 3rd party tries to physically extort them from me. In these cases, it should simply be technically impossible for me to obtain them. (If this 3rd party can bribe someone in the ISP, they might then be able to identify my visitor, which is unacceptable.) So that's basically my line of thinking. It's more for fun, of course, rather than the website being very sensitive. Call it paranoia :) Aug 12, 2012 at 16:15
  • But you have, in your answer, made a clear case against collecting IPs to track visits anyway. So I guess I'll just forget IPs anyway. I'll look into sessions instead. However, is there any way to view the visitor's country without using IPs? Aug 12, 2012 at 16:16
  • @SecurityClown You could use the browser's locale, but that's quite unreliable. Most people will have their browser in their language, but many powerusers use English versions instead (like myself, I'm from NL but use an English OS and browser). How about looking up the location based on the IP, then discarding the IP (or hashing it or so) and only saving the location?
    – Luc
    Aug 12, 2012 at 17:25
3

Let's consider the whole state of your systems. That state can be duplicated; just shutdown the machines and grab a copy of the disk contents.

Now get that copy as initial state to some virtual machines, which will emulate your servers. Power them up, and do the accounting thing which yields the number of "unique visits". This gives a number N. Then emulate a connection from a given IP (these are virtual machines in a virtual network, so you can emulate the whole Internet as you wish), and do the accounting again. If this yields N+1, then the IP was new; otherwise, it will return N, revealing that the IP was already accounted for.

This extremely generic method shows that regardless the amount of cryptography you may throw at the problem, you, as the system administrator, still have the power to test whether a given IP is in the set of "previously seen IP addresses" or not. Given that there are only four billions or so of possible IP addresses (in IPv4), exhaustive search is feasible. That's a power you cannot give up as long as your machines are yours. And if you cannot lose this power, you will not be able to convince anybody that you don't have it (except gullible people).

At best, you can use slow hashing (like bcrypt) to make the exhaustive search harder, but you cannot raise the bar too high because your server will have to do this hashing for every incoming client. Also, all these hashes will have to use the same "salt", because this is not the same exact situation as password hashing. Anyway, you will not be able to prove that you do not keep IP addresses somewhere, because it would be easy to plug another machine which passively spies on the network and records IP addresses from incoming connections.


As was mentioned by @Luc, IP addresses are not a secret. They come from a limited set which can be exhaustively enumerated with existing technology. A different situation is when you want to track email addresses, to detect duplicates, but without actually keeping them in your database because you don't want to be the guy responsible for leaking millions of addresses to any spammer who steals a backup copy. For that situation, hashing is convenient. The main difference between IP addresses and email addresses is the size of the set of possible addresses, making exhaustive enumeration highly feasible for IP addresses and totally ludicrous for email addresses.

2

I confirm you that is possible not recording the ip addresses of visitors. As an example, if you are using apache as web server you can customize the log format so that it does not track the ip address. In this way you don't need hashing ip addresses.

However if you have to track unique visitors of your site, using ip for that purpose does not help. Try to think to a LAN with undreds of potentials visitors. You should use instead the session mechanism and store the unique session id coupled with the actual date instead of ip.

Altought this cannot assure you the ip addresses are not stored elsewhere like in routers or firewalls or other sw/hw devices.

4
  • @SecurityClown consider also to store your data such that they are not accessible from the internet. if you want to keep them really secured. Please rate answers!
    – GiovaLomba
    Aug 11, 2012 at 6:59
  • Sessions would be another option for sure... Interesting. Aug 11, 2012 at 7:53
  • I'm assuming you'd store the session ID in a cookie or something on the client end?
    – Rohan
    Aug 11, 2012 at 8:39
  • @RohanDurve-Decode141 Durve - Decode141 That was not the subject of the question? Does it?
    – GiovaLomba
    Aug 11, 2012 at 10:31
1

http://en.wikipedia.org/wiki/Bloom_filter - Bloom Filter allows you to make groups (filters) and add data to it, which is non-reversible due to false positives.

Other methods are not OK, because it's easy to attack 32bit IPv4 space.

1

If you concern about the privacy of your visitors but you want to have analytics check piwik. http://piwik.org/privacy/

you can hide the ip like 201.xxx.xxx.xxx (I do not know if the same apply in the database of the piwik installation)

In the past I have saw a technique with piwik and apache to not log ip addresses but log browsers ID or something like that. I am searching the place but I can not found it.

0

I have heard of people hashing the IP address and other information (user agent string, browser locale ...) and storing that hash. As long as it isn't just the IP address this should make it impossible to discern the data from the hash, yet retain the option to cluster the visitors.

You didn't give any details, but for Apache there is a module named removeip (Debian/Ubuntu package for Apache2: libapache2-mod-removeip).

Problem in the latter case is that you can't tell who is who. But then, any information that you can use to track your users can be used by people who extort that information from you. So as you pointed out already: best method is not to have that data at all in the first place.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .