7

I use rsync (also called deltacopy) to a copy a large (6GB) text file (SQL database dump) on to a remote server. this gets me about 100:1 bandwidth reduction over simply copying the file as most of the content us unchanged day-to-day

This is a single file which contains all the data in the database, rsync compares the content piece by piece with the destination content (using hashes) and only sends the hashes and the changed parts down the wire.

I can instead produce a dump with one file per database table, but the Only tables likely to be unchanged day-to-day are small.

I want to start encrypting this file at the source, and storing only encrypted copies at the destination, but it seems to me that any encryption stronger than a simple substitution cypher is going to compromise the efficiency gain I get from using rsync.

Is there some strong encryption that I can use for this purpose?

1
  • Is there anything else you would like me to add to my answer?
    – forest
    Apr 23 at 21:13

2 Answers 2

4

You can use duplicity for that. Duplicity is an application written in Python which uses librsync, GnuPG, and tar for incremental, encrypted, and signed backups. It is specifically designed for situations where the server is untrusted (for example, when Dropbox or an Amazon VPS is used). Duplicity calculates the delta between the local archive and the remote archive (with metadata cached locally) and sends an encrypted file to the remote server containing records of incremental changes and associated data. Each snapshot file is typically 25 MiB in size (although that is configurable), and without the key, it is impossible to tell if the snapshot file contains 25 MiB worth of new files, 25 MiB of incremental changes, or even 25 MiB of metadata, like file deletion records.

Duplicity will generally only transmit a single, full backup once, and then all subsequent backups will be in the form of deltas. This makes it similar to a revision system where all past changes can be reverted. When a local file is deleted and this is synced to the remote server, the remote file is simply marked as having been deleted. Of course, it is possible at any time to "clean" old changes and do a new full backup in order to save space. See the manual page for more information.

The benefit of duplicity is that the destination server can be completely untrusted without compromising the ability to do efficient, incremental backups. A moderate cache of metadata must be stored locally in order to make this possible, but you can trade-off the storage requirement for bandwidth requirements and re-download the metadata before each sync.

6
  • I don't think duplicity will work, the input is a single text file. (or a tar format database dump which will have one file per database table. and all tables will have changes) I'll edit the question to make this clearer.
    – Jasen
    May 16, 2018 at 4:16
  • @Jasen I believe duplicity supports incremental per-file backups. By default this is limited to 25 MiB increments, but it can be changed to the average size of a changed table.
    – forest
    May 16, 2018 at 5:26
  • the database records are mostly 50 to 5000 bytes. each record can be any size (2G max) . some records will have changes most will not.
    – Jasen
    May 17, 2018 at 2:40
  • How much do you expect to change between backups?
    – forest
    May 17, 2018 at 3:14
  • maybe 1% of records. I'm getting like 100 times speed-up using rsync vs scp
    – Jasen
    May 18, 2018 at 1:57
0

An encryption algorithm is considered effective when its output cannot be differentiated from random data. Thus it's not possible to have both good compression and encryption - as compression relies upon identifying patterns of data.

Your solution is most likely to only encrypt and transfer the difference between versions of that file.

7
  • This is incorrect. As I pointed out in my answer, you can do client-side encrypted backups to an oblivious server and still provide compression, deduplication, and incremental changes.
    – forest
    May 16, 2018 at 3:14
  • It's not incorrect that the purpose of encryption is to make data random - and compression relies on patterns, the opposite of random. Your answer describes a tool which does what I suggest in my answer, encrypt the difference between versions of a file.
    – P. Cap
    May 16, 2018 at 3:29
  • 2
    Obviously it is not possible to compress already encrypted data, but that is not what OP is asking about. In fact, OP is not asking about encryption at all, but securely transmitting encrypted deltas.
    – forest
    May 16, 2018 at 3:32
  • But, I'm not compressing the file, or am I?
    – Jasen
    May 17, 2018 at 2:36
  • it seems storing deltas will make it hard to discard old data.
    – Jasen
    May 17, 2018 at 2:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.