0

In a nutshell, I need:

  • to hash a file and to make this hash public, to be able to prove data integrity when sharing the data privately with a trusted third party.
  • to choose the right hashing method to prevent plausible brute force attacks by an attacker who know some important features of the data. I guess it involves using some salt/pepper methods, but I'm a newbie and not sure about the right method: hmac(data, key), sha256(concat(data+salt)), or something completely different?

Here's my detailed use case:

I have some survey data containing personal information, for example under the form of a csv file:

id Age Gender City Income Children
1 24 M Buenos Aires 35,000$ 0
2 36 F New York 120,000$ 2
3 ... ... ... ... ...
250 11 M Paris n/a n/a

etc. (that's just an example, some surveys may have more simple structures, others more complex).

I cannot share the raw data publicly for privacy reasons, even if I can and want to share it privately with a handful of trusted third parties (let's call them party A).

I want to generate a hash of the data, and publish this hash on a third party public server (github or whatever) at time X. The goal of publishing the hash is to be able to prove later to party A that the data have not been manipulated or tampered with between time X and now.

I may not necessarily know who is party A at time X, so simply sharing the data or the hash privately with them at time X is not an option.

Also, I don't want an attacker to be able to "reverse engineer" the data, assuming they'll know how the data is structured -i.e. they may know all the existing column names in the csv file, as well as all the possible values inside each column. For example, they know for sure there's an "Age" column in the data, and that all the "age" values lie between 0 and 90 (but they don't know how old a specific person is in the dataset, they just know she's aged between 0 and 90).

It's my understanding that just hashing the data and making the hash public make the data susceptible to a brute force attack (the attacker could generate a set of plausible csv files, hash them, and compare all these generated hashes to the public hash until finding a match). As aggregate data will be made public (e.g. information like "90% of men in the data live in New York and have an income between 50,000$ and 150,000$"), I guess it would make a brute force attack much more easier.

From what I've read on the subject, I understand that I should use a "salt/pepper" approach, where I'd add a private salt to the data and generate the hash from that, and then privately share the salt with trusted party A.

But I'm not a security specialist at all, so I'm not sure which specific method is the best to do that. For example, I've read this discussion https://stackoverflow.com/questions/64325456/is-it-appropriate-to-use-an-hmac-and-pepper-to-hash-input-from-a-limited-domain, but I'm not sure at all if hmac would be actually appropriate for my use case. Many resources about salting seem to talk about passwords, but I'm a bit wary that the advice I've read does not apply to data files.

Any idea or suggestion? I'm a bit new to this kind of problem, so I'm very comfortable with hearing that I'm completely off track with this.

If it can somehow help answering my question, I work under Windows (job obligation), generally with Python, but I've no problem using other programming languages or tools if necessary.

Thanks.

5
  • Did you do any estimation how long a brute force attack would take?
    – Josef
    Mar 17, 2022 at 15:27
  • Not at all! I stumbled upon this problem today, I'll try to test it tomorrow. Not sure if I can do it easily. Just as an additional information, the type of data I work with are almost always categorical (i.e. each column takes generally less than 10 different values; it's variables like "Gender", "educational level", etc.), which certainly makes a brute force attack easier than datasets containing continuous values (like the "income" column in my example).
    – J-J-J
    Mar 17, 2022 at 15:37
  • @JosefsaysReinstateMonica given Ben Voigt's answer, I don't think I'll try to empirically estimate it, it seems to be practically impossible to brute force even very simple survey data.
    – J-J-J
    Mar 17, 2022 at 17:36
  • @kelalaka thanks for the reference! I guess I have some additional reading to do before getting a good grasp of my issue. That will teach me to be too optimistic too quickly. Hopefully Ben Voigt's answer will still be useful for my case, particularly the part relative to adding random garbage rows to the data.
    – J-J-J
    Mar 17, 2022 at 17:47

1 Answer 1

1

If you don't hash individual records, but only the complete file containing some tens of thousands of rows, the brute force attack is infeasible. Even of all the variables were boolean, and each row contained only 6 columns, the search space for a file of 10,000 rows is pow(2, 60000).... and we consider encryption keys of length 256 bits to be unguessable. The attacker is going to find a hash collision long before he finds the true dataset.

Just use a crypto-quality hash (one with strong preimage and collision resistance) and a large enough input. You don't need a keyed HMAC. You don't need salt.

4
  • Thanks, that's reassuring! As it's not my area of expertise, I prefered to be overcautious than not enough cautious. Just to be clear, the number of rows in the datasets I work with can generally be counted in hundreds, or at best thousands. Tens of thousands is extremely rare (I often work with social scientists, who work on a "small" scale compared to big private companies). Does it change something? Thanks anyway!
    – J-J-J
    Mar 17, 2022 at 16:06
  • @JJJ: As long as the number of cells (rows * columns) is in the thousands the guy trying to brute force it is going to have a bad day/month/year/lifetime/heat-death of the universe. If you're worried about your data being too short, add some (thousands of) extra rows of random garbage generated from a crypto-quality randomness source aka CRNG. As long as that randomness is part of the hash function input, the attacker has to brute force it as well. You just need to make sure you remember the bogus data too, because your hash doesn't prove anything without it.
    – Ben Voigt
    Mar 17, 2022 at 16:14
  • (Ok, concatenating your data with a strong random secret is a simple form of HMAC)
    – Ben Voigt
    Mar 17, 2022 at 16:17
  • Thanks again for the very clear and useful advice, I think it saved me a lot of time.
    – J-J-J
    Mar 17, 2022 at 16:21

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .