I am looking for tools/libraries for symmetrically encrypting millions of files (many terabytes' worth of content).

In particular, I am exploring the possibility of using gpg2 for this1.

One concern I have is that many years may elapse between the time these files are encrypted and the time it becomes necessary to decrypt one of them. Therefore, it is essential to preserve whatever is required for decryption all this time. I am not clear of what this would be in the case of gpg2.

This example will clarify my point.

When I type the following on my terminal

% gpg2 --symmetric > the_answer.gpg

...a dialog pops up asking for a passphrase, and after I enter it, another dialog pops up asking to confirm it2. After I do all this, I can type in a message. So I continue with

% gpg2 --symmetric > the_answer.gpg

...and then I type Ctrl-D. At this point, the file the_answer.gpg contains an encrypted version of the message I typed in.

If I now run gpg2 --decrypt the_answer.gpg, gpg2 just goes ahead and decrypts the message immediately, without asking for a passphrase at all:

% gpg2 --decrypt the_answer.gpg                                               
gpg: AES256 encrypted data
gpg: encrypted with 1 passphrase

I figure that gpg2 must have stored the passphrase somewhere3.

Q1: If, say, 10 years in the future, I had the file the_answer.gpg, plus the original passphrase, and some other machine running some future version of gpg2, would this be enough to decrypt this file, or would I need something else? Or would I also need to have some persistent database-type files currently residing somewhere in my computer?

Q2: Since gpg2 implements an open standard (OpenPGP), are there other tools besides gpg2 that I could use to decrypt the_answer.gpg if I know the passphrase?

1 ...even though I realize that gpg2 is a tool optimized for asymmetrically encrypting an individual's email communications, a very different task from what I have in mind.

2 I realize that such dialogs are a deal-breaker if one wants to encrypt millions of files. I understand that it is possible to supply the password without going through these dialogs, so I will ignore them for the purpose of this question.

3 Moreover, somehow it knows that this passphrase was used for this message. Indeed, if I encrypt other messages with different passphrases, I can decrypt them too, as shown above, without ever being prompted for a passphrase.

  • 1
    Re 3: GPG has long supported an 'agent' process, which remembers your passwords (or decrypted privatekeys) from one command to another, usually across all shells/sessions/desktop(s) etc on a system. Running gpg-agent used to be a separate step, though your system's packaging/install might do it for you; in current versions gpg itself does it automatically. How to stop the agent varies depending on lots of things too complex for this comment, but shutdown and reboot definitely will do it and then you'll need to re-enter the password(s). – dave_thompson_085 Jan 10 at 10:53
  • 1
    Also re terabytes: it weakens your encryption very slightly if a file exceeds a smallish fraction of the 'birthday bound', which for 64-bit block ciphers means maybe 32GiO/1000. Above that for maximum safety use a 128-bit block cipher, which currently is only AES{,192,256} Twofish or Camellia. (Twofish was a nonwinning AES candidate, and Camellia is basically an AES competitor.) – dave_thompson_085 Jan 10 at 10:59

Yes, to both of your questions.

GPG uses the passphrase to programatically generate a symmetric encryption key that is used for the encryption/decryption process when the --symmetric flag is passed to the program.

Yes, any program that uses the OpenPGP algorithms can create the decryption key from the same passphrase (that algo is the special sauce, and it is part of the open code, so anyone can use it).

Also, just so ya' know, PGP/GPG encrypts everything with symmetric encryption. The difference triggered by the --symmetric flag is that normally it generates a completely random encryption key per output file, but then uses asymmetric encryption to encode this key to each listed "encrypted to" party, and includes those at the beginning of the file. That way any of the encrypted-to parties can use their private key and the first part of the file to find the symmetric decryption key. But since symmetric is so much more efficient, it is what is actually used to encrypt the payload data.

  • But one thing no one will ever know or be able to answer with any amount of safety: Whether those standards / tools will still be around in X years of whether the support for GPG will be dropped with the Windows version released at year XY. But yea, in case it'll be like this, just decrypt with the latest tool / windows version available, look for another tool and encrypt again and hope for long-term support for yet another 20 years :) – tim Jan 9 at 15:38
  • Good point @tim . One should expect to be able to decrypt the files in 10/20/200 years time, but one should not expect to have a new version of GPG in 2219 to perform the decryption. As long as you have any working version of the software (and the necessary hardware/OS to run it) then you can decrypt. Heck, at my office we have a main line-of-business application that left post-end-of-life extended support 15 years ago. The program continues to work (without any issue for a dozen years) and we have people on staff who know it as well as the original programmers did. Not new, but works. – Ruscal Jan 9 at 15:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.