0

Normally, to upload an encrypted file I would first encrypt the entire file, then upload the encrypted file. However, for large files this is not that easy as I need a external hard drive to hold the encrypted file. I am wondering if it is possible to perhaps encrypt a large file, say of 500 GB size, bit by bit, and to send the bit by bit encrypted information to an upload stream to say, Google Drive? Thanks.

5
  • 1
    Zip files can split files into chunks for this very reason
    – schroeder
    Nov 7, 2019 at 22:41
  • 1
  • If I were to split, would it be the case that I must create the split files all at once, hence making it so I must have partitioned space for all the zip files? Or can I create a small split zip file, upload it, delete it, then move onto the next split zip?
    – user321627
    Nov 8, 2019 at 0:04
  • @user321627 That depends entirely on what tool you use to split. Some tools literally do nothing other than to cut the file into smaller parts.
    – user163495
    Nov 8, 2019 at 10:39
  • 1
    This is more about superuser? What is the security question here?
    – kelalaka
    Nov 8, 2019 at 10:50

1 Answer 1

1

Yes. Your main problem would be that you will need a client that allows receiving the file from a stream, not as a single file.

For instance, if you were copying the file through ssh, you would normally¹ do something like

scp myfile.txt remotehost:

However, this expect the prior file to exist. But it so happens that this would be equivalent² to:

cat myfile.txt | ssh remotehost 'cat > myfile.txt'

which is a form where you can now easily insert an encryption filter

cat myfile.txt | gpg -e -r AA00BB00CC00DD | ssh remotehost 'cat > myfile.txt.gpg'

Another alternative would be to use a virtual filesystem that showed the files as encrypted, but perform the encryption on access. This would make the encryption layer transparent to the programs that perform the upload.

An example of this would be the encfs utility, that uses a FUSE filesystem. The man page itself shows an example of using it with the --reverse flag in order to create an encrypted view for copying the files.

Finally, a last method to not need the double disk space, would be to encrypt the file over the plaintext one. That is, use a program that encrypts the first 10MB³ of the file and replaces the first 10MB of the original file with the encrypted ones. Then the second chunk of 10MB and so on.

This bears the risk however that if the process failed mid-way (such as due to a power loss), you would get a half-crypted file. And if you wanted to keep the file decrypted, you would need to perform a second pass inline decrypting the file after the upload. I would only recommend it if you wanted to leave the local file encrypted but you have no space to process it differently. It still solves your problem, though,

' I would consider the most common cli commands to be scp, sftp and rsync, all of them presenting the same issue.

² For normal files, not taking metadata into account

³ Or any other arbitrary chunk size


Update: OP comment asks whether it would be possible to break the file in different parts, encrypting and uploading them separately., which is obviously possible. You only need to go in turn extracting pieces of the file, encrypting and uploading them.

The following script would just do that:

#!/bin/bash
FILENAME=file.dat  # File to encrypt, set to "$1" if using as a command
KEY=user321627  # Identifier of the PGP key to use
BLOCKSIZE=$((10 * 1024 * 1024))  # Use 10 MB chunks
FILESIZE=$(stat -c %s "$FILENAME")
BLOCKS=$(( ($FILESIZE - 1 ) / $BLOCKSIZE + 1))
for i in $(seq 1 $BLOCKS); do
    echo Encrypting part $i of $BLOCKS
    dd if="$FILENAME" bs=$BLOCKSIZE skip=$(($i - 1)) count=1 |       gpg -e -r $KEY > "$FILENAME.$i.gpg"
    sha256sum "$FILENAME.$i.gpg" >> hashes.txt
    echo Uploading part $i
    upload-file "$FILENAME.$i.gpg"
    rm "$FILENAME.$i.gpg"
done

Adjust as needed. I am assuming that encryption is done with gpg to an OpenPGP key (generally a good choice), and that there is a command upload-file that sends the file passed as parameter to your final destination, such as Google Drive.

Given that you probably also want integrity in addition of confidentiality, I am storing a hash of each encrypted part on hashes.txt so that you can later verify each piece.

2
  • Thanks! Suppose I wanted to break the file into 10 parts, is it possible to encrypt the first part, upload it, delete it, then encrypt the second part, without having encrypted all 10 parts first? In other words, is there a way for me to encrypt parts of a file without having encrypted the others?
    – user321627
    Nov 8, 2019 at 3:48
  • Sure. See the above update
    – Ángel
    Nov 11, 2019 at 0:15

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .