Tracking IP address to prevent abuse without logging user metadata

Question

I am trying to balance good security practices against excessive logging of user metadata / Personally Identifiable Information.

I am constructing a web app that allows for secure messaging. In part of my system design, I am trying to minimize leaking of user metadata.

Part of my system design includes a module that tracks IP addresses to prevent abuse, such as Denial-of-Service or account cracking. I only keep the log of IP addresses around as long as needed. A straightforward approach would be to log the IP address and a timestamp in a logfile, and delete entries after a period of time. To head off any distractions, the intent is to ban abusive IP addresses for a set amount of time (i.e., linear offset), not permanently.

The threat model is that the logs could be used to determine who was using the system and when. I want the users of this system to have confidence that even if a server is compromised, that the logs are not designed in such a way that a third party could infer who used the system, and when, to talk to whom.

My first thought was that storing information in a form that can be verified later, yet not stored in plaintext, is similar to how one securely stores system passwords using a key derivation function (i.e., applying a hash to a passphrase and a salt for a large number of iterations). Thus, the same way that a user password can be verified against its KDF-based hash, an IP can be checked against its KDF-based hash in the log file to see if the same IP address has been logged before.

What are the strengths and weaknesses of this approach? Are there superior methods for storing user metadata / PII not in plaintext, in a format that a web app can verify?

Updated to clarify web app purpose, and the threat model.

Answer 1

Lets say you need to view the IPs over a 30-minute period to decide on abuse. Here is a scheme to keep the IPs no longer than one hour.

Every 30 minutes, generate a memory-only random string, RS.

For any time period, keep the current and previous RS: rs_c, rs_p.

For every IP, compute the current hmac using the current random string:

IP_c = HMAC(rs_c, IP)

Store the IP in a table:

IP_c, last-update

Compute the ip count over the last hour.

func get_ip_count(ip):
  IP_p = HMAC(rs_p, IP)
  IP_c = HMAC(rs_c, IP)
  count = [select count(*) from ipHistory where IP in (IP_p, IP_c) and last-update < [30-minutes-ago]]

As long as you can protect the RS, you'll not recover the IPs for more than your sliding window.

Clean the table based on last-update/age.

delete from ipHistory where last-update < [30-minutes-ago]

Answer 2

(This may be a dupe; I have a vague memory of a similar question, but I can't find it now, so either my memory or my search-fu is defective, or it's been deleted.)

Salt only works if you can tie the protected value to a public value (i.e. salt for hashing a password is stored with the corresponding userid). The only obvious match to an IPaddress is the DNSname which is at least as sensitive and not always 1-to-1 anyway. Pepper (fixed but secret) is still useful against an attacker who gets logfile data but not the app that writes it. But there's an inherent limitation: you can recognize a duplicate hash, but you can't easily go back the IPaddress if you want it for something like firewall rules, you can only do hash-based checks in apps you write yourself.

For a more flexible solution, you could public-key encrypt the values with deterministic padding (which the standard paddings are NOT, precisely to prevent recognizing duplicates) and provide the privatekey only (and temporarily) to admin apps that need to decrypt -- perhaps run on a different (safer) machine.

But any scheme that detects duplicates can be bruteforced by an attacker who gets the logfile and the pepper or publickey (or the app that contains either), since the IPaddress space (for v4, and 99.99% of the public net is still v4) is somewhat less than 32 bits.

Which brings up another approach -- separate the sensitive data from the compromise risk. Instead of writing the logfile on the web server, which must be publicly accessible and is usually complicated enough to have vulnerability risks, send it to another machine which is firewalled to run only a dead-simple write-only logging program -- maybe 20 lines of code that can be rigorously checked.

Edit: clarifications for comment, 2015/02/24.

Salt itself doesn't need to be public (although sometimes it is), but it needs to be determined by public data, and unique per item with at least high probability if not certainty. The canonical example is password hashing, where both the usually-random salt and the salted hash of the password are stored in a database accessed by userid. The whole point of salt is to be different for different values. A value added to the hash that is secret (in your case to the app server) but the same for all hashed values is pepper. That is useful, as I said, if an attacker can get data containing the peppered hashes but not the pepper itself.

I contrasted public-key encryption as "flexible" in how you can use the results. For example a filter program could take a given IPaddr, such as a new connection, and hash it (with the pepper) to see if it matches a logged entry, but it can't take a logged entry or several and (cheaply) determine the IPaddr(s). Public-key encryption can be reversed to show the IPaddr by only a specific program or (possibly remote) system to which you give the private-key, and protect and restrict accordingly.

Answer 3

A straightforward approach would be to log the IP address and a timestamp in a logfile, and delete entries after a period of time.

Straightforward but phenomenally innefficient. Really you should only be maintaining data for violators in indexed storage independent of your log data (you should probably also be keeping log files but that is seperate discussion).

Since I presume you don't feel as strong a duty of care to the violators as you do to other people using your site, this solves both the performance issue and the privacy problem.