I think the only thing you should keep in mind is that, while the header forces the browser to download the file, this doesn't prevent the asset from being included inside an actual page if a malicious user finds a way to inject it. As a for instance, if the download URL was something like this:
And the server let references to it be injected directly into another page:
CSP settings. As a result, XSS injection becomes more dangerous.
In reality though the biggest risk (from what I understand about the system you have described) is probably legal issues (note: IANAL). If anyone can upload anything to the server, I imagine there are potential legal concerns to be aware of. The server you describe could effectively be used to distribute any information/files/content desired by anyone who has access to it: especially if the service doesn't require authentication. What if happens if someone decides to use it to distribute child pornography? That's an extreme example, but sometimes such things are unfortunately relevant, and there are plenty of less extreme examples that still cause problems.