6

Imagine the following scenario:

  • Files are kept on a Linux server and clients access them via Samba.
  • One of the client machines gets infected with ransomware, which encrypts everything within the logged-on user's reach, including files on the server's shares.

I am curious as to whether there are any solutions (apart from regular backups) which would allow for full and easy recovery from such attacks.

A rough design I had in mind would be based on some kind of snapshotting/version management:

  • Whenever a file is changed or deleted, keep a copy of the old version around somewhere.
  • These previous versions would be read-only for ordinary users, protecting them against tampering by any kind of ransomware running on client machines.
  • Previous versions would be available via a special path, e.g. a previous version of /home/john/path/to/file.odt would be in /home/john/path/to/~snapshot/20161120_163242/file.odt.
  • Some kind of mechanism to revert entire shares to an earlier state.
  • Optionally, heuristics which detect suspicious activity (e.g. large numbers of files being read, new files of similar size being created and the original files being deleted, or large-scale modification of files) and trigger appropriate actions (alerting the admin, blocking write access).
  • Protection should not rely solely on particular characteristics of known ransomware – while these might work against those particular trojans, they could easily be totally useless against future ransomware that is implemented in a different way.

Is there any software that does this kind of thing, runs on Linux and is FOSS?

  • 3
    Check out FreeNAS backed by ZFS. ZFS allows you to take periodic snapshots of the filesystem. If a ransomware attack occurs, you simply revert to a previous snapshot. – tlng05 Nov 20 '16 at 20:15
  • Thanks, that was a good pointer. With periodic snapshots, I'd be somewhat worried about lots of redundancy – but digging down a bit deeper, the copy-on-write approach found in ZFS and Btrfs is indeed the core of what I had in mind. – user149408 Nov 20 '16 at 20:55
  • I've had some bad performance problems with btrfs in the past. Make sure its good enough for you before using it on a production machine. – Out of Band Nov 21 '16 at 19:15
  • @Pascal can you elaborate (e.g. what kind of operations are problematic, how badly does performance decrease)? With a file server, the bottleneck tends to be the network connection, thus a moderate decrease in filesystem performance might not even have a real impact. – user149408 Nov 22 '16 at 14:50
  • I never took exact measurements. Like you I was taken in by the cow properties of the file system. An older version of Ubuntu used it to handle recovery after aborted system upgrades (e.g. by taking a snapshot beforehand and rolling back if necessary) and the system became very sluggish as a result, much more so than a comparable system which used ext4 or a slightly older one with reiserfs. I traced it back to btrfs not performing well under high load, but like I said, that was in the past (about 24 - 18 months ago), so it might have been a problem with a specific btrfs version. – Out of Band Nov 22 '16 at 15:39
2

I think backups are your only sure bet. What you suggest as a solution basically consists of making a special backup; I'd suggest you just keep several (at least two) full backups around. The problem is then reduced to detecting ransomware working on your files and recovering from the last good backup. You can use deduplicating backups to save space. The advantage of going the backup route is that

  1. you introduce less complexity (no additional system just for ransomware)
  2. you improve the quality of your backup routine - once you're automating alerts and recovery, you've put in place a lot of stuff to make backup recovery a breeze. That's good for a much broader category of failures than just ransomware.

As for detecting ransomware, that's easy: Encrypted files all look like random data, so you can simply do statistical tests for each file (equal distribution of bytes? not compressable using zip? etc) and count the number of files that look random.

1

Currently your main enemy is Locky:

enter image description here

This article has details on using Fail2Ban to do what you want by detecting locky ransom notes:

[Definition]

failregex = smbd * \.. \ IP = <HOST> \ | * \ locky $.

. Smbd * \. \ IP = <HOST> \ | * _ Locky_recover_instructions \ .txt $
  • 1
    Thanks, in fact the Heise article mentioned in your source inspired me to ask this question – it would work against Locky but not against tomorrow's crypto-ransomware. I'm looking for a more generic solution (have since amended the question), one that'll also provide "recovery" and an optional action for the following: mount the share, cd to it, then run compress -r . – which behaves very similar to Locky and its friends. – user149408 Nov 20 '16 at 20:04
  • Thanks for clarifying the question. Always too bad the answers look less related after edits :). This might also be interesting: unixmen.com/performing-incremental-backups-using-tar – J.A.K. Nov 20 '16 at 22:44
0

I’ve read up on a few things since, and came up with the following design—not yet tested in practice, but it would presumably do what I want it to.

Background

The “enabling technology” behind this setup is Btrfs, of which we harness three features:

  • Copy-on-write (COW): When files are copied, Btrfs initially does not duplicate the payload—both files point ot the same region on disk, and a true copy is created only if one of the two files is altered. This makes duplication of files very easy and lightweight.
  • Subvolumes: These are volumes within a volume. Subvolumes are inherently mounted with their parent volumes (appearing as regular directories) or mounted in their own right.
  • Snapshots: A snapshot is a copy of a subvolume, and in itself a subvolume. Due to the COW nature of Btrfs, the overhead is minimal. If the subvolume being snapshotted has child subvolumes, these will not be included (if this is desired, snapshot them individually). Snapshots can be read/write or read-only. In the latter case, they are immutable even if they are part of a read/write mount.

Setup

  • Create a btrfs volume to hold all file shares. (As FHS designates /srv as the root of service-specific user data, the mount point could be something like /srv/samba or /srv/file.) Mount it with the noatime option (this ensures that simple reads do not alter the file’s metadata, further minimizing the overhead of snapshots).
  • For each file share, create a Btrfs subvolume. They can have the same mount attributes as their parent and thus do not need to be mounted separately.
  • Run a periodic (nightly, hourly—depending on your needs) cronjob which creates read-only (important!) snapshots of the subvolumes, as subvolumes of the latter. For example, today’s daily snapshot of /srv/file/joe goes in /srv/file/joe/snapshot/daily-20180317. Optionally the script can delete snapshots which are past a certain age. A sample can be found here.
  • If possible, take the following extra actions:
    • Separate privileged accounts (e.g. those with sudo permissions) from accounts used to access network shares
    • Restrict the ability of ordinary users to run commands on the system (disallow shell logins or restrict commands to the bare minimum).
    • Lock the root account (enforcing the use of sudo).
    • Require the user’s password for privileged operations.

Mechanism of Action

A ransomware tool encrypting everything within its reach would not simply be able to encrypt the snapshots. It would need to:

  • Be aware it is running against a network share (there are known crypto-trojans which encrypt cloud storage as well, so this is not a real obstacle).
  • Be aware that the files are being served off a Btrfs disk, and be able to detect its subvolume structure.
  • Escalate its privileges on the server to mess with Btrfs subvolumes (e.g. to delete snapshots or replace them with read-write ones)—this is our main reason for separating file share user accounts from privileged ones.

If done correctly (i.e. at least one of the three preconditions is expensive enough for an attacker), that raises the bar significantly.

If malware strikes, the following is going to happen:

  • Files may get encrypted but the snapshot remains unaffected. Recovery point objective is equal to the time between shapshots, i.e. at most, modifications made since the last snapshot will be lost. This can be countered by increasing the snapshot frequency (e.g. going from daily to multiple times a day, hourly or even multiple times an hour).
  • Due to the COW nature of Btrfs, encrypting a file will take up extra space. If the disk is near capacity, it may run out of space before the encryption finishes. This may prevent encryption of the remaining files (the exact behavior depends on the trojan, so don’t count on it).

Extras

Back up your system regularly—snapshots merely complement this. While snapshots are helpful if you need to recover from data corruption caused by faulty software, user errors or certain malicious actions, they are useless if your disk goes bad.

This setup could be further complemented by detecting suspicious activity and triggering countermeasures.

Indications of suspicious activity are:

  • Files being overwritten on a large scale, seemingly systematic (though some trojans go only after certain file types)
  • High entropy in content of overwritten files (simple test: space saved on compression is close to zero—though some trojans encrypt only parts of the file, which would be harder to detect this way)
  • Changes to files you wouldn’t normally expect to change (like the holiday photos from a few years ago, or even honeyfiles created just for this purpose—less effective, though, if the trojan happens to encrypt these files only after having encrypted most of the rest)

Potential actions include:

  • Trigger an alert so the admin can look into it.
  • Block write access to the share, for the user, from the suspicious IP, for the entire server, for a group of servers, for all servers as you deem appropriate—that will stop the trojan from continuing its work, at the expense of blocking some legitimate uses.
  • If blocking write access is impractical, throttle the bandwidth to the server (possibly selectively as well)—this will at least slow down the encryption process while allowing legit usage to continue, albeit with reduced performance.
  • Disable snapshot rotation—if it is an attack and it has been going on for some time, you may not be able to recover everything from the latest snapshot but may need to go back in time.
  • If you have forensic procedures in place, trigger those as appropriate. That will most likely include procedures to preserve evidence, such as disabling log rotation or enabling more detailed logging.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.