To protect personally identifiable information (PII) in a SQL database, we have implemented field-level encryption. However, we need to allow fulltext search against some of those data.
One idea that I was considering was creating a Lucene search index of the unencrypted values, but ensuring that all of the fields containing PII are indexed but unstored.
The process would work like this:
- The user submits e.g., a partial national ID number in a search form in the application.
- The application submits the query to Lucene.
- Lucene returns a collection of documents. The documents do not contain any PII; rather each document contains the PK value of the corresponding record in the SQL database.
- The application retrieves the corresponding records from the database and decrypts the values to display to the user.
However, this leaves open the question, if someone were to gain read access to Lucene's index files, could they reconstruct the unencrypted values?