Safely downloading user submitted content

Question

TL;DR We save the request payload and Content-Type header in a database. It should be JSON or CSV but may be malformed. Support users download the request payload as a file in their browsers using Content-Disposition: attachment. We attempt to add a file extension of either .json or, the default, '.csv'.

How can the support users download this file safely when it could contain absolutely anything? I'm less worried about ZIP bomb type attacks but more spyware, ransomware, etc., etc.

Background

We have a REST API that supports both JSON and CSV content types.

The API is secured with a unique client certificate for each user.

For reasons of arbitration we must keep an audit record of the exact content submitted to the service, we do this by storing the bytes directly in the database along with the content type header and some other fields from the URL and client certificate. All of this, with the exception of the certificate is pre-validation as we need to record invalid submissions as well.

We have a support team a very activity for them is to help users who are not able to submit their documents because of a) malformed content or b) the content violates some business rule.

Now a requirement has come in to allow our support team to download the audit information so they can try to reproduce the users issue by submitting it to a test system or just looking at the content to spot mistakes.

We will provide a link to the file with either a .csv or .json extension if we can detect the content type.

The problem

Obviously this is wide open to XSS from the point of the unvalidated fields but that is mitigated easily enough by correctly encoding it for display however I do not see how this is possible for the downloaded file.

For example, I submit an Excel spreadsheet with Content-Type: text/csv, a support user downloads the file and opens in Excel to view the CSV and gets warned "the file extension doesn't match the content, do they want to open anyway?" to which they click "yes" because they are trying to figure out why it's malformed and my malicious spreadsheet is opened.

Even if we remove the file extension requirement then a support user may try to open it in Excel anyway seeing the content is supposed to be a CSV.

Solutions

I cannot think of any robust solutions;

• Train support users to open in plain text editor first, but users are fallible)
• Prevent downloading of content that is not character data, but character data can still be an attack vector, i.e., a .bat file and doesn't meet the requirements
• Encode content and display on screen for copy/paste thus ensuring it's only text, files could be large and currently use streaming from DB to output stream to keep memory overhead low.
• Trust our users and that they keep their client certificates safe

Is it just that the requirements will always be at odds with security and there is no way to download user submitted files safely?

Answer 1

IMHO, your key problem is that you are not validating the content being uploaded. Since we are only talking about JSON and CSV here, its trivial to write code to validate these. It won't catch all the issues, but it will get most of them. And at least you can tell the uploading user they've made a mistake at the time they are uploading - rather than telling a content consumer sometime later that they can't have the content.

As a minimum I would suggest validating the character set and the structure of the file - and checking it matches the submitted mimetype. However there is a complication in that not all the things which are thought of as CSV comply to RFC 4180, and RFC 4180 actually describes a lot of variants of the CSV file format (particularly how the symbols used for meta-data are represented when they occur within data).

I would strongly advise against allowing anyone to download content which has not been checked via their browser. Since it should be possible to represent the content using UTF-8 or UTF-16, its trivial to create a representation of the content with the non-compliant characters replaced and displayed in a PRE.../PRE block within an html page (with the htmlentities translated too).

Answer 2

I am going to assume that your support workers will download these files using a web browser.

If you are going to display the file inline in the browser:

1. You should disable content sniffing. A malicious user could try to upload a script with a fake content type, and some browsers try to "sniff" the actual content type and run it. Set the X-Content-Type-Options: nosniff response header on your server

2. You want to host the user content on a totally different domain name than the site that you use to get it. For example, google uses googleusercontent.com for user uploaded data for gmail.com . This would prevent an evil upload from stealing or over-writing your cookies on your domain

Else:

1. Set your server to send the file as an attachment, not inline, so that a malicious script cannot be run in the browser. Content-Disposition: attachment; filename="filename.csv" response header

2. If IE browser, you could use a custom protocol to force the file to be opened in specific application (say Notepad). See https://stackoverflow.com/questions/10990484/execute-local-command-with-javascript-in-chrome/10992181#10992181

Both:

1. You could use libraries on your server to validate that the uploaded documents are well-formed. Jackson for JSON and Super CSV for CSV with Java

Answer 3

As I understand your question, the actual problem is:

Users can upload arbitrary bytes and then trick my help desk people into opening that data as a file in Excel.

What if, for example the bytes contain not a csv but a excel file with a macro virus?

What can I do?

Well. You could

• have excel not installed on their workstations
• restrict execution of macros
• check for plausibility on the server ("is this plain text? Is it csv or json parsable?"), throwing an error else.
• as the submissions are signed, bill your customer that submitted it for all outages resulting from abuse or attack.