Can I use RSA to encrypt an AES256 key that I put at the beginning of my AES256 encrypted file?

Question

I have a master server generating large files for other (known) servers to download.

Those files:

• Are quite big. From 20MB to 5GB.
• Are written once by the server and then it forgets about them.
• Nobody should be able to read the content of the file except the server it has been generated for.
• Will be downloaded over HTTPS using a library such as RequestJS

Based on this knowledge, I planned to generate an RSA key pair for each server, store the public key in the database of the master server along with the servers' specific data (like unique identifier) and only keep the private key on the server it belongs to.
I thought I would just have to encrypt the whole 5GB file using the server's public key and only it would be able to decrypt it.

However, after looking around on the internet, I noticed that it's an extremely bad idea to encrypt big things using RSA. It's not meant for this purpose at all.

I also heard a lot about AES and decided to take a look at the way the TLS works and noticed it uses RSA encryption for the symmetric AES keys.
Now, I'm thinking that I should encrypt my files using AES with a randomly generated key per file but I need to transmit the key to the server.

My idea is the following:

• Generate a random 256 bits key per generated file.
• Use this key to encrypt the whole file.
• Use the public key of the target server to encrypt this key.
• Prepend the encrypted key to the file so that it's a whole package.
• Sign the whole thing with the master private key (but I haven't thought about that yet so I'm open to suggestions regarding the algorithm)

And, on the reception side, the server would just read the first 256 bits, decrypt with its private key and use the decrypted payload as a key to decrypt the file itself.

Since I have no deep knowledge of security, I would like to be sure that it a good idea. I see no real downside but I may not know what I'm talking about enough to judge that.

At first I thought HTTPS would be sufficient but I don't want anyone to be able to get the file so I still need more encryption than the connection itself.
Moreover, I wouldn't want a "slave" server to be able to decrypt the content of another. And I should be able to revoke a server by changing its key pair.

Answer 1

You are reinventing the wheel by trying to solve your problem at a low level. You are trying to combine RSA and AES encryption primitives to create something that will asymmetrically encrypt a file. What you should be doing is looking for a solution that can asymetrically encrypt a file. For example,

• NaCl crypto box
• GPG

Answer 2

What you've defined in your process is called Key Encapsulation. It's how TLS functions as well as GPG and many other cryptographic data exchange protocols.

GPG is particularly easy to demonstrate, since there is an option to list detailed information regarding what is contained within the PGP message. This option is --list-packets, and along with a little information from the standard, RFC4880, it's fairly easy to dissect an encrypted PGP message.

Here are the applicable areas of RFC4880:

4.3.  Packet Tags

   The packet tag denotes what type of packet the body holds.  Note that
   old format headers can only have tags less than 16, whereas new
   format headers can have tags as great as 63.  The defined tags (in
   decimal) are as follows:

       0        -- Reserved - a packet tag MUST NOT have this value
       1        -- Public-Key Encrypted Session Key Packet
       2        -- Signature Packet
       3        -- Symmetric-Key Encrypted Session Key Packet
       4        -- One-Pass Signature Packet
       5        -- Secret-Key Packet
       6        -- Public-Key Packet
       7        -- Secret-Subkey Packet
       8        -- Compressed Data Packet
       9        -- Symmetrically Encrypted Data Packet
       10       -- Marker Packet
       11       -- Literal Data Packet
       12       -- Trust Packet
       13       -- User ID Packet
       14       -- Public-Subkey Packet
       17       -- User Attribute Packet
       18       -- Sym. Encrypted and Integrity Protected Data Packet
       19       -- Modification Detection Code Packet
       60 to 63 -- Private or Experimental Values
...

5.1.  Public-Key Encrypted Session Key Packets (Tag 1)

   A Public-Key Encrypted Session Key packet holds the session key used
   to encrypt a message.  Zero or more Public-Key Encrypted Session Key
   packets and/or Symmetric-Key Encrypted Session Key packets may
   precede a Symmetrically Encrypted Data Packet, which holds an
   encrypted message.  The message is encrypted with the session key,
   and the session key is itself encrypted and stored in the Encrypted
   Session Key packet(s).  The Symmetrically Encrypted Data Packet is
   preceded by one Public-Key Encrypted Session Key packet for each
   OpenPGP key to which the message is encrypted.  The recipient of the
   message finds a session key that is encrypted to their public key,
   decrypts the session key, and then uses the session key to decrypt
   the message.

---

So, what we're looking for is a packet with Tag 1: Public-Key Encrypted Session Key Packet, which contains the encrypted symmetric key that was used to encrypt message.

Let's generate an Elliptic Curve25519 test key and experiment:

gpg --expert --full-gen-key

pub   ed25519 2019-01-05 [SC]
      8142894DE02523BAC0B830518D1D7EACEC02106D
uid           [ultimate] Demo Key 2 (Delete Me) <demo2@domain.tld>
sub   cv25519 2019-01-05 [E]

Here's a listing of the generated key:

gpg --list-key --keyid-format long demo2@domain.tld

pub   ed25519/8D1D7EACEC02106D 2019-01-05 [SC]
      8142894DE02523BAC0B830518D1D7EACEC02106D
uid                 [ultimate] Demo Key 2 (Delete Me) <demo2@domain.tld>
sub   cv25519/7B2F2DF3D9ABA877 2019-01-05 [E]

Now create an encrypted message:

echo -e "Hello There.\n" | gpg -o message.gpg -er demo2@domain.tld

Let's look at what's in the encrypted message:

gpg --list-packets message.gpg

gpg: encrypted with 256-bit ECDH key, ID 7B2F2DF3D9ABA877, created 2019-01-05
      "Demo Key 2 (Delete Me) <demo2@domain.tld>"
# off=0 ctb=84 tag=1 hlen=2 plen=94
:pubkey enc packet: version 3, algo 18, keyid 7B2F2DF3D9ABA877
    data: [263 bits]
    data: [392 bits]
# off=96 ctb=d2 tag=18 hlen=2 plen=73 new-ctb
:encrypted data packet:
    length: 73
    mdc_method: 2
# off=117 ctb=a3 tag=8 hlen=1 plen=0 indeterminate
:compressed packet: algo=2
# off=119 ctb=cb tag=11 hlen=2 plen=20 new-ctb
:literal data packet:
    mode b (62), created 1546740749, name="",
    raw data: 14 bytes

Yes, indeed there is a packet with Tag 1. This packet contains an encrypted symmetric session key which was used to encrypt the message.

It's useful to note that there is very little effective difference between the process of how GPG encrypts messages and how TLS functions at the cryptographic level. The protocol is a bit different and the trust anchors are different, but the mechanism used to encrypt the data and decrypt it at the far end are nearly the same. In both cases, there is a symmetric session key that is used to encrypt the message. In both cases, the symmetric message key is encrypted with the public key of the receiver and sent along with the encrypted message itself.

It's clear that your described algorithm matches what is routinely done programmatically by many modern cryptographic messaging systems.

So, yes GPG satisfies the requirement on the transmission side. However, how can you be sure that the transmitted message or file has not been replaced?

And this is where the server signatures on the file become important. Many software developers have adopted the method of generating a SHA256SUM hash of the distributed file and then signing the hash sum list with the developer private key. The signed sums are distributed via TLS enabled web pages. This completes the loop. The receiver of the file can now ensure that the received file is the same file distributed by the server by checking the downloaded file's hash against the server displayed and signed hash list.