1

TL;DR We save the request payload and Content-Type header in a database. It should be JSON or CSV but may be malformed. Support users download the request payload as a file in their browsers using Content-Disposition: attachment. We attempt to add a file extension of either .json or, the default, '.csv'.

How can the support users download this file safely when it could contain absolutely anything? I'm less worried about ZIP bomb type attacks but more spyware, ransomware, etc., etc.

Background

We have a REST API that supports both JSON and CSV content types.

The API is secured with a unique client certificate for each user.

For reasons of arbitration we must keep an audit record of the exact content submitted to the service, we do this by storing the bytes directly in the database along with the content type header and some other fields from the URL and client certificate. All of this, with the exception of the certificate is pre-validation as we need to record invalid submissions as well.

We have a support team a very activity for them is to help users who are not able to submit their documents because of a) malformed content or b) the content violates some business rule.

Now a requirement has come in to allow our support team to download the audit information so they can try to reproduce the users issue by submitting it to a test system or just looking at the content to spot mistakes.

We will provide a link to the file with either a .csv or .json extension if we can detect the content type.

The problem

Obviously this is wide open to XSS from the point of the unvalidated fields but that is mitigated easily enough by correctly encoding it for display however I do not see how this is possible for the downloaded file.

For example, I submit an Excel spreadsheet with Content-Type: text/csv, a support user downloads the file and opens in Excel to view the CSV and gets warned "the file extension doesn't match the content, do they want to open anyway?" to which they click "yes" because they are trying to figure out why it's malformed and my malicious spreadsheet is opened.

Even if we remove the file extension requirement then a support user may try to open it in Excel anyway seeing the content is supposed to be a CSV.

Solutions

I cannot think of any robust solutions;

  • Train support users to open in plain text editor first, but users are fallible)
  • Prevent downloading of content that is not character data, but character data can still be an attack vector, i.e., a .bat file and doesn't meet the requirements
  • Encode content and display on screen for copy/paste thus ensuring it's only text, files could be large and currently use streaming from DB to output stream to keep memory overhead low.
  • Trust our users and that they keep their client certificates safe

Is it just that the requirements will always be at odds with security and there is no way to download user submitted files safely?

  • Are the files supposed to be confidential? – A. Darwin Mar 31 '16 at 18:55
  • As you can see from the answers, your question is not quite excellent in its wording. Making downloads them self safe is easy, with content-disposition. It is not absolutely clear what you are asking. If I guessed correctly, please try to shape your wording to better fit your actual question. – Tobi Nary Mar 31 '16 at 19:49
  • @A.Darwin - files are not confidential as support users can see all details within successfully submitted content anyway. – James Apr 1 '16 at 7:52
  • @SmokeDispenser - added a TLDR to hopefully simplify. Was trying to give as much detail as possible originally but maybe that made it more confusing than it needed to be. – James Apr 1 '16 at 7:53
  • 1
    @James If they are not confidential, there are some online platforms which you can use to automatically check files against a wide variety of antivirus-antimalware solutions. For example, you could ask VirusTotal for their private API (I'm not affiliated with them, just used the standard website a couple of times). You should be able to automatically (or semi-automatically) upload files to the VirusTotal platform, which apparently checks them using >50 different products (see virustotal.com/en/faq/#difference-private-api). – A. Darwin Apr 1 '16 at 8:35
2

IMHO, your key problem is that you are not validating the content being uploaded. Since we are only talking about JSON and CSV here, its trivial to write code to validate these. It won't catch all the issues, but it will get most of them. And at least you can tell the uploading user they've made a mistake at the time they are uploading - rather than telling a content consumer sometime later that they can't have the content.

As a minimum I would suggest validating the character set and the structure of the file - and checking it matches the submitted mimetype. However there is a complication in that not all the things which are thought of as CSV comply to RFC 4180, and RFC 4180 actually describes a lot of variants of the CSV file format (particularly how the symbols used for meta-data are represented when they occur within data).

I would strongly advise against allowing anyone to download content which has not been checked via their browser. Since it should be possible to represent the content using UTF-8 or UTF-16, its trivial to create a representation of the content with the non-compliant characters replaced and displayed in a PRE.../PRE block within an html page (with the htmlentities translated too).

  • I still have to record the content as exactly what was submitted for contractual reasons, however for download I can validate that it is UTF-8 compliant like you say and this is the solution I will go for. – James Apr 4 '16 at 9:25
  • If you are contractually obliged to receive and store malware, you might want to think about how you draw up such contracts (and encrypt known bad data). – symcbean Apr 4 '16 at 10:12
  • it's for arbitration, e.g., customer disputes that they successfully uploaded the information, we have to be able to say, no you didn't and it was because it was this data. But yes, the contract is not ideal and there are many scenarios where the information never reaches that far such as incorrect URL/missing cert where the same dispute could be had. One of the joys of being a long way from the real "customer" in a chain of organisations collaborating on a project. – James Apr 5 '16 at 11:09
1

I am going to assume that your support workers will download these files using a web browser.

If you are going to display the file inline in the browser:

  1. You should disable content sniffing. A malicious user could try to upload a script with a fake content type, and some browsers try to "sniff" the actual content type and run it. Set the X-Content-Type-Options: nosniff response header on your server

  2. You want to host the user content on a totally different domain name than the site that you use to get it. For example, google uses googleusercontent.com for user uploaded data for gmail.com . This would prevent an evil upload from stealing or over-writing your cookies on your domain

Else:

  1. Set your server to send the file as an attachment, not inline, so that a malicious script cannot be run in the browser. Content-Disposition: attachment; filename="filename.csv" response header

  2. If IE browser, you could use a custom protocol to force the file to be opened in specific application (say Notepad). See https://stackoverflow.com/questions/10990484/execute-local-command-with-javascript-in-chrome/10992181#10992181

Both:

  1. You could use libraries on your server to validate that the uploaded documents are well-formed. Jackson for JSON and Super CSV for CSV with Java
  • Thanks for the tips, as one of the problems the support users might be investigating is malformed, but not malicious, CSVs and JSON it's a bit tricky. Do you know if it's possible to ensure is textual only? I'm worried about windows being helpful and detecting something as executable like in the case with excel opening CSV files that are actually spreadsheets. – James Apr 1 '16 at 8:50
1

As I understand your question, the actual problem is:

Users can upload arbitrary bytes and then trick my help desk people into opening that data as a file in Excel.

What if, for example the bytes contain not a csv but a excel file with a macro virus?

What can I do?

Well. You could

  • have excel not installed on their workstations
  • restrict execution of macros
  • check for plausibility on the server ("is this plain text? Is it csv or json parsable?"), throwing an error else.
  • as the submissions are signed, bill your customer that submitted it for all outages resulting from abuse or attack.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.