I’ve read up on a few things since, and came up with the following design—not yet tested in practice, but it would presumably do what I want it to.
Background
The “enabling technology” behind this setup is Btrfs, of which we harness three features:
- Copy-on-write (COW): When files are copied, Btrfs initially does not duplicate the payload—both files point ot the same region on disk, and a true copy is created only if one of the two files is altered. This makes duplication of files very easy and lightweight.
- Subvolumes: These are volumes within a volume. Subvolumes are inherently mounted with their parent volumes (appearing as regular directories) or mounted in their own right.
- Snapshots: A snapshot is a copy of a subvolume, and in itself a subvolume. Due to the COW nature of Btrfs, the overhead is minimal. If the subvolume being snapshotted has child subvolumes, these will not be included (if this is desired, snapshot them individually). Snapshots can be read/write or read-only. In the latter case, they are immutable even if they are part of a read/write mount.
Setup
- Create a btrfs volume to hold all file shares. (As FHS designates
/srv as the root of service-specific user data, the mount point could be something like /srv/samba or /srv/file.) Mount it with the noatime option (this ensures that simple reads do not alter the file’s metadata, further minimizing the overhead of snapshots).
- For each file share, create a Btrfs subvolume. They can have the same mount attributes as their parent and thus do not need to be mounted separately.
- Run a periodic (nightly, hourly—depending on your needs) cronjob which creates read-only (important!) snapshots of the subvolumes, as subvolumes of the latter. For example, today’s daily snapshot of
/srv/file/joe goes in /srv/file/joe/snapshot/daily-20180317. Optionally the script can delete snapshots which are past a certain age. A sample can be found here.
- If possible, take the following extra actions:
- Separate privileged accounts (e.g. those with
sudo permissions) from accounts used to access network shares
- Restrict the ability of ordinary users to run commands on the system (disallow shell logins or restrict commands to the bare minimum).
- Lock the root account (enforcing the use of sudo).
- Require the user’s password for privileged operations.
Mechanism of Action
A ransomware tool encrypting everything within its reach would not simply be able to encrypt the snapshots. It would need to:
- Be aware it is running against a network share (there are known crypto-trojans which encrypt cloud storage as well, so this is not a real obstacle).
- Be aware that the files are being served off a Btrfs disk, and be able to detect its subvolume structure.
- Escalate its privileges on the server to mess with Btrfs subvolumes (e.g. to delete snapshots or replace them with read-write ones)—this is our main reason for separating file share user accounts from privileged ones.
If done correctly (i.e. at least one of the three preconditions is expensive enough for an attacker), that raises the bar significantly.
If malware strikes, the following is going to happen:
- Files may get encrypted but the snapshot remains unaffected. Recovery point objective is equal to the time between shapshots, i.e. at most, modifications made since the last snapshot will be lost. This can be countered by increasing the snapshot frequency (e.g. going from daily to multiple times a day, hourly or even multiple times an hour).
- Due to the COW nature of Btrfs, encrypting a file will take up extra space. If the disk is near capacity, it may run out of space before the encryption finishes. This may prevent encryption of the remaining files (the exact behavior depends on the trojan, so don’t count on it).
Extras
Back up your system regularly—snapshots merely complement this. While snapshots are helpful if you need to recover from data corruption caused by faulty software, user errors or certain malicious actions, they are useless if your disk goes bad.
This setup could be further complemented by detecting suspicious activity and triggering countermeasures.
Indications of suspicious activity are:
- Files being overwritten on a large scale, seemingly systematic (though some trojans go only after certain file types)
- High entropy in content of overwritten files (simple test: space saved on compression is close to zero—though some trojans encrypt only parts of the file, which would be harder to detect this way)
- Changes to files you wouldn’t normally expect to change (like the holiday photos from a few years ago, or even honeyfiles created just for this purpose—less effective, though, if the trojan happens to encrypt these files only after having encrypted most of the rest)
Potential actions include:
- Trigger an alert so the admin can look into it.
- Block write access to the share, for the user, from the suspicious IP, for the entire server, for a group of servers, for all servers as you deem appropriate—that will stop the trojan from continuing its work, at the expense of blocking some legitimate uses.
- If blocking write access is impractical, throttle the bandwidth to the server (possibly selectively as well)—this will at least slow down the encryption process while allowing legit usage to continue, albeit with reduced performance.
- Disable snapshot rotation—if it is an attack and it has been going on for some time, you may not be able to recover everything from the latest snapshot but may need to go back in time.
- If you have forensic procedures in place, trigger those as appropriate. That will most likely include procedures to preserve evidence, such as disabling log rotation or enabling more detailed logging.
