I am trying to balance good security practices against excessive logging of user metadata / Personally Identifiable Information.

I am constructing a web app that allows for secure messaging. In part of my system design, I am trying to minimize leaking of user metadata.

Part of my system design includes a module that tracks IP addresses to prevent abuse, such as Denial-of-Service or account cracking. I only keep the log of IP addresses around as long as needed. A straightforward approach would be to log the IP address and a timestamp in a logfile, and delete entries after a period of time. To head off any distractions, the intent is to ban abusive IP addresses for a set amount of time (i.e., linear offset), not permanently.

The threat model is that the logs could be used to determine who was using the system and when. I want the users of this system to have confidence that even if a server is compromised, that the logs are not designed in such a way that a third party could infer who used the system, and when, to talk to whom.

My first thought was that storing information in a form that can be verified later, yet not stored in plaintext, is similar to how one securely stores system passwords using a key derivation function (i.e., applying a hash to a passphrase and a salt for a large number of iterations). Thus, the same way that a user password can be verified against its KDF-based hash, an IP can be checked against its KDF-based hash in the log file to see if the same IP address has been logged before.

What are the strengths and weaknesses of this approach? Are there superior methods for storing user metadata / PII not in plaintext, in a format that a web app can verify?

Updated to clarify web app purpose, and the threat model.

  • There are publicly available libraries for mitigating abuse. Note that IP addresses are largely designed to be public, but only really have correlation to the physical world if you know the layout of the network. Which ISPs are (understandably) rather loathe to share. This isn't helped by the fact that a lot of consumer IPs are dynamic, and thus change, or may be hidden behind some form of NAT. And DOS attacks may lie about their origin IP address. What exactly are you trying to protect your users from? – Clockwork-Muse Aug 18 '14 at 7:46
  • Thanks for your note CM. Tried to update text to clarify what the web service does, and what I'm trying to protect against. – taltman Jan 19 '15 at 2:16

Lets say you need to view the IPs over a 30-minute period to decide on abuse. Here is a scheme to keep the IPs no longer than one hour.

Every 30 minutes, generate a memory-only random string, RS.

For any time period, keep the current and previous RS: rs_c, rs_p.

For every IP, compute the current hmac using the current random string:

IP_c = HMAC(rs_c, IP)

Store the IP in a table:

IP_c, last-update

Compute the ip count over the last hour.

func get_ip_count(ip):
  IP_p = HMAC(rs_p, IP)
  IP_c = HMAC(rs_c, IP)
  count = [select count(*) from ipHistory where IP in (IP_p, IP_c) and last-update < [30-minutes-ago]]

As long as you can protect the RS, you'll not recover the IPs for more than your sliding window.

Clean the table based on last-update/age.

delete from ipHistory where last-update < [30-minutes-ago]
  • Thank you for your reply. I stated KDF instead of hash because I want to have an adjustable work-effort (i.e., repeated hashing). By stating HMAC, are you advising against KDF? I'm assuming that the example of "IP, last-update", is meant to be, "IP_c, last-update", correct? By stating that you're storing the current and previous period "salts" in-memory, you're stating that the web app should be persistent, and not on-demand like a CGI? While on the topic, why not store the entire log in memory, instead of using some sort of SQL back-end? – taltman Feb 21 '15 at 6:56
  • Yes, it should be IP_c and I just updated it. – Jonathan Feb 25 '15 at 22:10
  • KDF vs HMAC - apples to oranges. In this situation you would use HMAC(<key>, <ip>) where key is a 256bit random string. To find out if an IP had existed in the history you would need the <key> and the <ip> and to guess the key is going to take a lifetime or two. KDF makes simple keys (<key>=="passw0rd") more time consuming to guess, but KDF is overkill if your <key> is a 256-bit random byte array. – Jonathan Feb 25 '15 at 22:15
  • Third question - why not just use a log instead of SQL? No reason. You can use a log file, an in memory variable, or post the IP_c and IP_p to pastebin or twitter. They values are secure enough (one way hash with 256-bit token) that you only need to protect the current and previous 256-bit tokens. – Jonathan Feb 25 '15 at 22:20
  • Thanks for your comments. Based on a threat model of a server compromise, how much "defense-in-depth" does the in-memory key provide? Won't an attacker who has rooted a box have easy access to the process memory? – taltman Feb 27 '15 at 6:21

(This may be a dupe; I have a vague memory of a similar question, but I can't find it now, so either my memory or my search-fu is defective, or it's been deleted.)

Salt only works if you can tie the protected value to a public value (i.e. salt for hashing a password is stored with the corresponding userid). The only obvious match to an IPaddress is the DNSname which is at least as sensitive and not always 1-to-1 anyway. Pepper (fixed but secret) is still useful against an attacker who gets logfile data but not the app that writes it. But there's an inherent limitation: you can recognize a duplicate hash, but you can't easily go back the IPaddress if you want it for something like firewall rules, you can only do hash-based checks in apps you write yourself.

For a more flexible solution, you could public-key encrypt the values with deterministic padding (which the standard paddings are NOT, precisely to prevent recognizing duplicates) and provide the privatekey only (and temporarily) to admin apps that need to decrypt -- perhaps run on a different (safer) machine.

But any scheme that detects duplicates can be bruteforced by an attacker who gets the logfile and the pepper or publickey (or the app that contains either), since the IPaddress space (for v4, and 99.99% of the public net is still v4) is somewhat less than 32 bits.

Which brings up another approach -- separate the sensitive data from the compromise risk. Instead of writing the logfile on the web server, which must be publicly accessible and is usually complicated enough to have vulnerability risks, send it to another machine which is firewalled to run only a dead-simple write-only logging program -- maybe 20 lines of code that can be rigorously checked.

Edit: clarifications for comment, 2015/02/24.

Salt itself doesn't need to be public (although sometimes it is), but it needs to be determined by public data, and unique per item with at least high probability if not certainty. The canonical example is password hashing, where both the usually-random salt and the salted hash of the password are stored in a database accessed by userid. The whole point of salt is to be different for different values. A value added to the hash that is secret (in your case to the app server) but the same for all hashed values is pepper. That is useful, as I said, if an attacker can get data containing the peppered hashes but not the pepper itself.

I contrasted public-key encryption as "flexible" in how you can use the results. For example a filter program could take a given IPaddr, such as a new connection, and hash it (with the pepper) to see if it matches a logged entry, but it can't take a logged entry or several and (cheaply) determine the IPaddr(s). Public-key encryption can be reversed to show the IPaddr by only a specific program or (possibly remote) system to which you give the private-key, and protect and restrict accordingly.

  • Thank you for your reply. Not sure why the salt must be "public". I intend for the web app host to store a single randomly-generated salt for all KDF IP hashes. Why do you state that the public-key approach is "more flexible" than using a pepper with an off-host log? I like your last suggestion and will explore it further. – taltman Feb 21 '15 at 6:36
  • @taltman see edit for response longer than I could easily handle in comments. Cheers. – dave_thompson_085 Feb 25 '15 at 3:42
  • What do you mean precisely by "public"? Public as in listed on a website? Do you consider the salt values stored along with Unix account information on a server to be "public"? – taltman Feb 27 '15 at 6:28
  • Can you address the KDF suggestion in the question? It sounds like your notion of an off-host service providing the 'pepper' is not necessarily mutually-exclusive from using KDF. Thanks! – taltman Feb 27 '15 at 6:33

A straightforward approach would be to log the IP address and a timestamp in a logfile, and delete entries after a period of time.

Straightforward but phenomenally innefficient. Really you should only be maintaining data for violators in indexed storage independent of your log data (you should probably also be keeping log files but that is seperate discussion).

Since I presume you don't feel as strong a duty of care to the violators as you do to other people using your site, this solves both the performance issue and the privacy problem.

  • Thanks for your answer. Inefficient in what sense? I can imagine more than one. For preventing certain types of abuse, you cannot know a priori whether a visitor is a violator or an innocent user, such as password cracking or DoS. How would you avoid tracking innocent users completely in a separate data structure? If there is no way, then the violator data structure is dependent on the more comprehensive list. In this question, I'm stating that I'm striving for a trade-off between adequate security via logging, but not excessive accumulation of user data that might be used against them. – taltman Mar 29 '15 at 0:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.