Employ fscrypt to securely protect, from data-at-rest exposure, all files+data installed after an initial Linux system build?

Question

Our team's goal

We want to protect our sensitive data from being potentially data-at-rest exposed after we relinquish a Linux VPS (see below) provided by a hosting service. We're exploring simple(r) fscrypt usage to potentially solve this problem.

We are _not_ seeking impenetrable, "perfect" security. We're seeking a more (most?) reasonable approach that pragmatically and reliably applies the currently-most-effective industry solution(s).

The following "design" is our first attempt to reach this goal.

Is the following accurate, feasible, and reasonable?

Details

We employ Linux VPSes to self-host some of our software/services, including from providers DigitalOcean (DO), AWS, and GCP. Often these VPSes do not come natively encrypted. They all appear to have ext4-with-file-based-encryption support, simply because ext4 has apparently supported Linux filesystem encryption since kernel v4.1. Most of our VPSes currently run Ubuntu v18.04 LTS, which employs Linux kernel v4.15.

We like the fscrypt concept but are new to its usage and paradigms (to be clear, we are complete fscrypt newbies). Here's our initial, uneducated run-down of what we're considering and why. In this case we use the DigitalOcean (DO) example. DO calls VPSes "droplets."

1. Create a new DigitalOcean (DO) droplet with ext4 on its main, root filesystem ("rootfs" aka '/')
2. Lock rootfs with ecryptionion via fscrypt, encrypted with some key file(s), aka "locking/unlocking keys."
3. [fscrypt presumably presents no special problem for database files, or vice versa.]
4. Automatically unlock rootfs at boot time, by reading/applying keys from step (2) or some similar, fully-automated process.
5. [All files created prior to step (2) are unencrypted.]
6. [All files created after step (2) are encrypted.]
7. We use (install software, read/write data, etc) the droplet.
8. [Days/months/years go by.]
9. We stop using the droplet and remove all data/services from it.
10. We securely-erase (secure-delete?) the keys created in step (2).
11. We optionally secure-erase software+data (files and directories) created in steps (7) and (8).
12. We delete the droplet by DO "destroy"-ing it.

Potential benefit 1: post-VPS-deletion data security / "proper" deletion
Our team sleeps a little more soundly understanding that DO admins or other DO clients will have a harder time recovering our data created in steps (7) and (8), especially since DO (and other hosting providers) might keep data around for a while even after droplet deletion. (Granted, these DO references only mention historical backups retaining data, and these backups could still contain the fscrypt unlocking keys - we do not see a way around this.)

Potential benefit 2: "immediate data deletion"
Imagine a slightly different scenario than the one above: our team experiences a crisis, we're really short on time, and we need to immediately "delete" the data and feel confident no on can regenerate it. In this case, we simply securely-erase all the "unlocking key(s)" (including ones in backups/copies/replications) for the fscrypt-ed files. If no copies are kept (of the "locking file(s)"), only one secure-delete operation (for one or many files, in one storage place) is needed. And poof, no more data available.

Potential benefit 3: server-bootup "authority required"
Again, adjusting the above, numbered-bullet-list scenario: let's say that our above server (let's call it server A) must first get authorization form a FreeIPA (or some such auto-policy server; we're new to sysadmin-ing these kinds of systems, too) system in order to decrypt the "fscrypt unlocking keys" early in server A's bootup stages before it can proceed with the rest of the bootup. And optionally the FreeIPA system might first be required to check-in with a human somehow (maybe via authorization from a trusted mobile device, similar to Apple's 2FA) before the FreeIPA system can authorize server A to bootup.

Is all of the above accurate, feasible, and reasonable? If so...

Big obstacle. Our largest, current implementation concern: how to perform step (3), unlocking the rootfs at boot time? Are the fscrypt "unlocking keys" presumably stored on a non-rootfs mount/partition? Is there some other fscrypt or additional, related procedure that can facilitate this? This portion of the system design also plays into benefit 3, above.

Additional consideration(s).

Some files not protected. We're not encrypting/protection any files created prior to step (2) (above). Our response: so what? Maybe a malicious attacker gets a log file with some interesting data in it. And while this is not optimal, they won't acquire our primary data assets; and we find this entirely better than no protection at all for any file assets that happen to get stored on the rootfs (instead of an encrypted storage volume).

DigitalOcean (DO) claims to "scrub" destroyed droplets. Yes, that's their claim. However, what does AWS and GCP do? We plan to scale to many VPSes in AWS and GCP, possibly more than in DO. Further, our team has no way to validate that DigitalOcean is performing this "scrubbing" activity and to what degree. Additionally, this DO feature does not serve the additional potential benefits mentioned above. Finally: we think it good practice to primarily rely on vendor to implement proper security protocol. If we did, then we'd consider using more "cloudier"/outsourced solutions than what we're currently doing.

Less-attractive alternatives (to fscrypt-based design) we see thus far

LUKS on ("under") native rootfs mount

Possibly preferred, but we can not find a reasonable and reliable way to apply LUKS (or any lower-level, block-storage/disk, full-disk encryption (FDE)) to VPSes that are already built. However, our initial analysis suggests it's too cumbersome and/or not automate-able (for constructs we employ like Terraform and Ansible) even if we could make it work for a DigitalOcean droplet--which seems unlikely. (DigitalOcean engineers: if it's feasible, please message me.) Thus this option is not scalable for our group environments with a larger number of VPSes. Hence our attraction to relatively flexible, dynamic, post-VPN-build fscrypt "application" (per above) is far more attractive.

LUKS on non-rootfs or network-storage mount

Yes, DigitalOcean (and possibly AWS and GCP) has network-storage-mountable, LUKS-based "Volumes." However, ensuring all our various software-server apps always "point" to this Volume is a cumbersome and pragmatically-impossible task. We'll probably still employ DO Volumes, but we'd prefer that ALL of our installed files/data/software is protected after step (12), above. Otherwise we have to perform encumbering, continual audits to ensure every single little install update did not add anything to the (unencrypted) rootfs. This is arguably unmaintainable from an operationally-efficient perspective.

And it's fundamentally the same problem, best we can tell, if we try to install a separate, new filesystem on a local/native mount on/in the VPS disk/storage.

eCryptfs no longer practical

It's apparent author Michael Halcrow writes: "Because of performance and functionality issues (file name length, possibility of page cache inconsistency) eCryptfs shouldn't be used for anything any more."

Building our own hardware servers, possibly at our own facility

This of course takes our operations to a whole new level, a place we'd like to avoid until much further down the road.

Answer 1

We resorted to simply making a LUKS volume for /home and similar data-oriented mounts. Manually mounting after bootup is performed with a script (that the sysadmin manually runs), which requests a LUKS-decryption password from keyboard/stdin. This is of course a bit tedious, for every server with this rig needs manual attention on every reboot.

Nothing else seems/seemed sufficiently simple or reliable.

Alternatively, we're looking into Ubuntu-Server-installed-on-ZFS, and then potentially leverage ZFS's native encryption somehow. This is not a near-term effort, and my team is keeping tabs on the progress.

Other suggestions are of course welcome.