3

I would like to be able to do database look-ups on a set of data based on one of the encrypted fields.

My understanding of the best practice for accomplishing this is to store a hash of the data I want to look up in addition to the encrypted version.

If this were a password I was hashing, I would make sure to hash each value with a distinct salt, but since I won't know what record contains the hash I'm looking for in this case this doesn't seem feasible.

What are the best practices in this situation for protecting my data?

(p.s. one of the fields I need to search by is SSN, which seems like it would be one of the easier values to create a rainbow table for)

2

So first warning first:

If data is secure, it cannot be searched against, only checked against.

Now regarding searching: There is no good way to search encrypted or hashed fields, it will ALWAYS eat up CPU. There is a reason for this: Your data is in a one way street. It gets retrieved into memory, checked against, the in memory version destroyed, and the entry minus the data you can't send gets sent. This is for security purposes as well as attack proofing purposes. So then it begs the question "How do I search against it?" Literally an entry at a time. Because each hash or encryption should be unique and never repeatable unless checked against with the initial value, you have to retrieve every encrypted object, go through them until you get a match, and then return the id or pkey of that entry. Is this long, drawn out, annoying, and secure? Yes. Can this be done faster or better? Not really.

More Warnings:

NEVER STORE A RAINBOW TABLE! Those are easily cracked vulnerabilities(Don't believe me?)

NEVER STORE SALTS IN THE DATABASE! If the data as a whole gets stolen, this encryption is now worthless

If the data needs to be *truly secure**, hash it through a one way hashing algorithm(like bcrypt)

IF you MUST search against encrypted data, use a separate decryption program to decrypt the data, do the search, and send it back again in a compiled, standalone binary app. This way if the public facing server has an unwelcome visitor, they can't get that data at all, the key is still secure, and your data is safe.

  • I have a very large dataset, and it needs to be searched real time. How can I Best do this with as minimal exposure as possible, but still have a performance app? – ben schwartz Oct 6 '15 at 17:05
  • Sorry, I got autocorrected: performant – ben schwartz Oct 6 '15 at 17:07
  • Application or web app? – Robert Mennell Oct 6 '15 at 17:45
  • It's a web app. – ben schwartz Oct 6 '15 at 17:46
  • Then with a fast enough server it feels like real time. There's really no way around that if you want to stay secure. Other than that you'd need a standalone application/server to do those searches away from your web app to return to the user, and the user would have to understand that for security reasons this request is going to take awhile. – Robert Mennell Oct 6 '15 at 18:06
-1

An important point you've not addressed in your question is that you don't need to worry about collisions - i.e. if you have 2 different data items resulting in the index entry, then queries will return extra rows which you filter based on the decrypted values / unhashed key.

Yes, you can't use a per-record salt for hashing your data. But you could use the same salt for all records. This does potentially become a weak point in the overall security.

  • 1
    Thanks, but you've only restated my problem, not suggested anything... – ben schwartz Oct 5 '15 at 15:32
  • I agree that this does not offer a solution to the problem. – RoraΖ Oct 5 '15 at 16:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.