4

To protect personally identifiable information (PII) in a SQL database, we have implemented field-level encryption. However, we need to allow fulltext search against some of those data.

One idea that I was considering was creating a Lucene search index of the unencrypted values, but ensuring that all of the fields containing PII are indexed but unstored.

The process would work like this:

  1. The user submits e.g., a partial national ID number in a search form in the application.
  2. The application submits the query to Lucene.
  3. Lucene returns a collection of documents. The documents do not contain any PII; rather each document contains the PK value of the corresponding record in the SQL database.
  4. The application retrieves the corresponding records from the database and decrypts the values to display to the user.

However, this leaves open the question, if someone were to gain read access to Lucene's index files, could they reconstruct the unencrypted values?

2

According to an article from 2013, the answer to this question appears to be, "Yes, unless you don't mind your search index being basically useless.":

Terms are stored in our classic inverted list, for each term there is a list of documents and, optionally, the position within that document. It's quite possible, though tedious, to reconstruct a document from the list of terms in the index. They may not be easily readable by a human because the terms have been through the entire analysis chain; think of stemming, synonym substitution, any of the transformations, and more, listed here. While these can be hard to glance at and read, the document so reconstructed should be considered complete; sensitive information is available.

[...]

Naturally ... the question arises "can't we encrypt [the index files]"? Sure, we can. It's just that doing so leads to some surprising results, often making the resulting index next to useless. Here's why.

A decent encrypting algorithm will not produce, say, the same first portion for two tokens that start with the same letters. So wildcard searches won't work. Consider "runs", "running", "runner". A search on "run*" would be expected to match all three, but wouldn't unless the encryption were so trivial as to be useless. Similar issues arise with sorting. "More Like This" would be unreliable. There are many other features of a robust search engine that would be impacted, and an index with encrypted terms would be useful for only exact matches, which usually results in a poor search experience.

1

As Lucene doesn't claim to be secure against such an attack, you can't assume that it is. Perhaps it is secure against such an attack this release, but being that this isn't viewed as part of its feature set, that could change in any minor patch. So you have to assume that it isn't secure.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.