0

This question already has an answer here:

I'm looking for a file encryption algorithm (or library) that supports efficient random access to the cleartext. That is, given an encrypted file, I need to be able to repeatedly read arbitrary byte-ranges from the cleartext with as little overhead as possible.

The only thing like this that I've found is SQLCipher, but (as the name implies) it's a special-purpose plugin for SQLite that only supports encrypting SQLite database files. I need something general-purpose.

This clearly isn't rocket science: I imagine one would do it by breaking the file into fixed-size chunks and encrypting each chunk with a symmetric cipher like AES. I'd need to use a different key for every block (derived by combining a master key and the block offset?), and I'd want an in-memory LRU block cache for performance. But it's fiddly enough that I'd much rather use an existing library than design it myself and run the risk of getting something disastrously wrong.

If I did have to design this myself, my main question would be key derivation: is it safe to derive each block's key from a master key and the block offset -- something like SHA2(masterkey || offset) -- or do I need to generate a random key for each block? (If the latter, those keys will need to be stored in the file, which makes it a bit trickier to keep the file chunks on filesystem block foundaries

marked as duplicate by Steve, Mark, WhiteWinterWolf, RoraΖ, Xander Jul 27 '15 at 12:25

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • 2
    en.wikipedia.org/wiki/… – user49075 Jul 26 '15 at 16:56
  • That addresses how to get a key for every chunk (and I also realized that NaCl / libSodium takes a nonce value that could just be a chunk offset.) There's more to the implementation than that, though, and I'd still rather use a library than write it myself. – Jens Alfke Jul 26 '15 at 17:06
  • Have a look at Wiki:Disk encryption theory – StackzOfZtuff Jul 26 '15 at 19:40
0

I'm looking for a file encryption algorithm (or library) that supports efficient random access to the cleartext. That is, given an encrypted file, I need to be able to repeatedly read arbitrary byte-ranges from the cleartext with as little overhead as possible.

What you are asking is highly dependent on the encryption primitive. You can't have arbitrary access to the plaintext without decrypting the ciphertext on each access. To reduce overhead you can decrypt the whole file once, cache it, and use it afterwards or, depending on the encryption scheme, store the minimum required amount of parameters needed to decrypt the needed information. For example, (secure)block ciphers chain all previous blocks prior to the encryption of the current one. Thus it is sufficient to know the output of the previous one(and the key) in order to decrypt the current block. Knowing the length of the block, the initialization vector(if used) and the decryption key you can store the outputs of each block. Unfortunately, in the case of AES, this does not reduce storage requirements(as you will be storing data with the length of the whole plaintext). However, depending on the segments of the file being used you can estimate a usage probability and cache the plaintext only for those entries. Practically, it is difficult to implement due to the (very) likely different offsets of useful plaintext and blocks.

This clearly isn't rocket science: I imagine one would do it by breaking the file into fixed-size chunks and encrypting each chunk with a symmetric cipher like AES. I'd need to use a different key for every block (derived by combining a master key and the block offset?), and I'd want an in-memory LRU block cache for performance. But it's fiddly enough that I'd much rather use an existing library than design it myself and run the risk of getting something disastrously wrong.

A makeshift solution would look like:

makeshift_crypto.sh

#!/bin/bash
d1="enc/"
d2="dec/"
# Split the file into separate files of 1000 bytes each, suffixed by _bigFile
split -b 1000 bigFile.tar.gz --additional-suffix=_bigFile
# Encrypt each file using aes256 cbc(with salt)
for fl in *_bigFile; do $ openssl enc -aes-256-cbc -salt -in "$fl" -out "$enc""$fl"; done
# Decrypt each file
for fl in `ls "$enc"`; do openssl enc -aes-256-cbc -d -in "$enc""$fl" -out "$dec""$fl" 

If I did have to design this myself, my main question would be key derivation: is it safe to derive each block's key from a master key and the block offset -- something like SHA2(masterkey || offset) -- or do I need to generate a random key for each block? (If the latter, those keys will need to be stored in the file, which makes it a bit trickier to keep the file chunks on filesystem block foundaries

No, it is not safe, as getting the key from block i(i is an index to that block) would allow retrieving all remaining keys n - i where n is the number of keys(blocks) and 0 <= i < n. Key management complexity also increases with the number of blocks. It is safe to use a single key for all of them(as you would if you would encrypt/decrypt the file entirely).

  • Can you explain why using key+offset is less secure than just key? The brute forcer has less data to work with (only 1KB instead of the whole file)? – ps95 Jul 26 '15 at 18:39
  • It is less secure because the suggested solution relies on both generation and storage of a number of keys derived from a "main" key, using a possibly home grown algorithm which would come in contradiction with Kerckhoffs's principle. – Sebi Jul 26 '15 at 19:31
  • sure. But using the same key on all blocks, f(x)=x seems les secure than a key function f(x) = sha256(x+index). If the attacker guesses x the encryption is blown in both cases. But if the attacker is trying to guess x, then the latter is harder. – ps95 Jul 26 '15 at 19:36
  • The issue lies in the fact that key management becomes more difficult(we have as many keys as we have blocks) and also the key derivation algorithm is unspecified. – Sebi Jul 26 '15 at 19:37
  • (a) Decrypting the entire file certainly doesn't count as efficient random access. (b) Your shellscript doesn't show where the encryption keys come from, which is the tricky part. (c) Your point about key safety doesn't make sense to me since I was talking about deriving chunk keys from the master key, so there's no key management problem. – Jens Alfke Jul 27 '15 at 18:15

Not the answer you're looking for? Browse other questions tagged or ask your own question.