Is it possible to encrypt portions of a file for upload so that one does not need to encrypt the entire file first?

Question

Normally, to upload an encrypted file I would first encrypt the entire file, then upload the encrypted file. However, for large files this is not that easy as I need a external hard drive to hold the encrypted file. I am wondering if it is possible to perhaps encrypt a large file, say of 500 GB size, bit by bit, and to send the bit by bit encrypted information to an upload stream to say, Google Drive? Thanks.

Answer 1

Yes. Your main problem would be that you will need a client that allows receiving the file from a stream, not as a single file.

For instance, if you were copying the file through ssh, you would normally¹ do something like

scp myfile.txt remotehost:

However, this expect the prior file to exist. But it so happens that this would be equivalent² to:

cat myfile.txt | ssh remotehost 'cat > myfile.txt'

which is a form where you can now easily insert an encryption filter

cat myfile.txt | gpg -e -r AA00BB00CC00DD | ssh remotehost 'cat > myfile.txt.gpg'

‎

Another alternative would be to use a virtual filesystem that showed the files as encrypted, but perform the encryption on access. This would make the encryption layer transparent to the programs that perform the upload.

An example of this would be the encfs utility, that uses a FUSE filesystem. The man page itself shows an example of using it with the --reverse flag in order to create an encrypted view for copying the files.

‎

Finally, a last method to not need the double disk space, would be to encrypt the file over the plaintext one. That is, use a program that encrypts the first 10MB³ of the file and replaces the first 10MB of the original file with the encrypted ones. Then the second chunk of 10MB and so on.

This bears the risk however that if the process failed mid-way (such as due to a power loss), you would get a half-crypted file. And if you wanted to keep the file decrypted, you would need to perform a second pass inline decrypting the file after the upload. I would only recommend it if you wanted to leave the local file encrypted but you have no space to process it differently. It still solves your problem, though,

‎

' I would consider the most common cli commands to be scp, sftp and rsync, all of them presenting the same issue.

² For normal files, not taking metadata into account

³ Or any other arbitrary chunk size

---

Update: OP comment asks whether it would be possible to break the file in different parts, encrypting and uploading them separately., which is obviously possible. You only need to go in turn extracting pieces of the file, encrypting and uploading them.

The following script would just do that:

#!/bin/bash
FILENAME=file.dat  # File to encrypt, set to "$1" if using as a command
KEY=user321627  # Identifier of the PGP key to use
BLOCKSIZE=$((10 * 1024 * 1024))  # Use 10 MB chunks
FILESIZE=$(stat -c %s "$FILENAME")
BLOCKS=$(( ($FILESIZE - 1 ) / $BLOCKSIZE + 1))
for i in $(seq 1 $BLOCKS); do
    echo Encrypting part $i of $BLOCKS
    dd if="$FILENAME" bs=$BLOCKSIZE skip=$(($i - 1)) count=1 |       gpg -e -r $KEY > "$FILENAME.$i.gpg"
    sha256sum "$FILENAME.$i.gpg" >> hashes.txt
    echo Uploading part $i
    upload-file "$FILENAME.$i.gpg"
    rm "$FILENAME.$i.gpg"
done

Adjust as needed. I am assuming that encryption is done with gpg to an OpenPGP key (generally a good choice), and that there is a command upload-file that sends the file passed as parameter to your final destination, such as Google Drive.

Given that you probably also want integrity in addition of confidentiality, I am storing a hash of each encrypted part on hashes.txt so that you can later verify each piece.