How should I more securely store files obtained by mirroring a website? [closed]

Question

Questions: what methods, tools, techniques, and procedures are recommended for securely storing files obtained by mirroring a website? What I mean is that I am looking for commonly known and accepted practices that align with the use-case outlined below, even if there might be varying points-of-view on which practices constitute a best practice.

Background: I have endeavored to provide as many specific details as I can provide on my use-case, though I admit that I struggle to provide more specific detail because this is an area in which my knowledge is limited. Thus, I am asking this question to better direct any future efforts, research, etc. As outlined in this question, I sometimes mirror websites to gain information of value to due diligence investigations. I learned from my previous question that things are often more complicated than expected. This question reflects my interest in thinking about the security implications of actually storing what I get when I use wget, curl, phantomJS, etc. So, I welcome your thoughts on best practices, actions you suggest I take, or general precautions. For example, perhaps it is a best practice to only store such files on an air-gapped machine, to convert HTML to a different format, or to simply avoid any Javascript files because they pose an inherent threat. As a reference, I use Ubuntu Linux.

Goals:

• Retain text-based files for keyword and other types of searching
• Retain image files for visual and metadata evaluation
• To the best of my ability given the other two goals, store and interact with files in a secure manner

Filetypes encountered: these are the main filetypes I've encountered to date...

• Documents: PDF, RTF, DOC, DOCX, ODF, TXT
• Presentations: PPT, PPTX
• Spreadsheets: XLS, XLSX, CSV
• Images: PNG, TIF, JPG, GIF
• Web: HTML, CSS, JS, PHP

Answer 1

The file types you listed and the goals you presented result in a huge attack surface. Assuming a sophisticated adversary, this gives a vastly increased possibility for exploitation. PDFs and PPT/PPTX are particularly problematic. If you cannot limit yourself to far fewer file types, you will need to isolate your activities, either using privilege separation or remote virtualization.

Privilege separation

This is a solution in the case that you need to perform this activity on your local computer. While it would be difficult to individually sandbox every single application you will be using, you can create a new user on your computer with few privileges:

• Disable access to su and sudo from the new user, and do not use them as that user.
• Do not su to your lesser user from root to avoid TTY pushback attacks.
• Use iptables to disable network access for that user.
• Set resource limits to reduce the amount of damage an exploited application can do.
• Use Wayland instead of Xorg if possible, or Xorg with systemd-logind to run it as non-root.
• Enable and use the Secure Attention Key when finished and switching to a new session.
• Scan for and remove all unnecessary setuid or setgid files, as well as setcap files.
• Use an auditing framework like auditd to monitor potentially malicious activities.
• Apply general system hardening such as sysctl tweaks or hardening patches.

Depending on the level of sophistication of the adversary, this may not be enough, and even for an adversary of moderate capability, this is rather incomplete, but it is a starting point. As you have to assume (most likely correctly) that the applications you are using to access these files are vulnerable to arbitrary code execution, the question turns into How can I safely run untrusted code?, which is of course extremely broad.

Virtual Private Servers

An easier solution would be to use a VPS. You can run applications remotely on the VPS rather than on your own computer and interact with it that way. Even if the VPS is completely compromised, the attack surface area is reduced to that of your SSH client and your terminal, which is fairly small. As you will not be able to directly view images over SSH (at least not safely), you may want to convert them to a very simple (and difficult to exploit) image format before transferring it to your local computer and viewing it. An example of such a format would be the PPM pixel map format. This also works for viewing PDF files, as they can be readily converted into image files using various utilities.

The host of a VPS is able to access and modify anything on the VPS. If this is an issue for you (for example, if the files are extremely sensitive, or integrity is of utmost importance), you may not want to use a VPS. This is unlikely to be an issue, and as the website you have downloaded from is already (presumably) public, there should be no confidentiality issue. You can increase the confidentiality and integrity of the data by instead using a dedicated server, although that would be more expensive.

You should keep a local backup of these files, in the case that the VPS is shut down, so that you can restore it to another VPS later. The local files should be stored in an "inert" form that will not be susceptible to exploitation of any indexers or thumbnail generators you may have on your system. This can be done for example by putting the files in an archive such as tar.

As you will need to use command line utilities rather than graphical ones, you will need to find ways to access these files over SSH. Some examples which you can do remotely on a VPS:

• You can convert PNG, TIFF, JPEG, and (non-animated) GIF to PPM to safely view it locally.
• XLS/XLSX can be converted to CSV, which is fine on command line. There is a good vim plugin.
• RTF, DOC, DOCX, and ODF can be converted to images, which can be securely viewed locally.
• I am not aware of any way to view PPT/PPTX in command line, though you can analyze them.
• HTML, CSS, and JS can be viewed in a text editor or run on a remote text-based browser.
• PHP can be viewed in a text editor or executed using a command-line PHP interpreter.

Answer 2

Since you asked about storage, my answer would focus on storing potentially infected files, and not on working with potentially infected files (which is a very different issue).

All the file types you mentioned, including some images, have been used or still are being used for distributing malware. So each of them presents a potential threat if unvetted.

However you cannot get infected by simply downloading a document file, so storing even an infected file is usually safe until and unless you open it. Generally you only get infected when you open/view the document file, and sometime you'd only get exploited when you use vulnerable software. But with some formats (like DOC) you might get exploited even with up-to-date software if you're not careful.

Thus your main concern in storing those documents securely boils down to preventing accidental clicks on them which would open them. The easiest way to prevent this is to scramble them. Even basic XOR would do the job here. This would prevent accidental openings of such files and thus would be secure enough.