How is it possible to detect malicious files or web shell on a vps web server?
I used to use a shared hosting, and when I tried to upload any malicious code for testing it got deleted automatically. Now I'm on my own vps and I want to increase its security. Is there any tool that can help me do this?
There's no single tool that can be used to detect everything with high certainty: there will be both false positives and false negatives, and thus some manual work and understanding is required. Also, we can't know what methods your former shared hosting was using. We can only tell you need layered security, where detecting malicious contents are just one measure among many.
This answer concentrates on finding backdoors, but gives some hints for prevention, too. The lists aren't comprehensive, although there are many perspectives. Also, for the layered security, you need to plan recovery and forensics: to learn more and to prevent the same from happening again. Many of the references I use in this answer are concentrating on WordPress, but the same principles applies to any other CMS.
Tools. Jonas Lejon: Finding PHP and WordPress Backdoors using antivirus and Indicator of Compromise lists some tools that could be used. It notices that virus definions from ClamAV aren't that good for finding PHP backdoors. VirusTotal might find some, but it's not for scanning whole sites nor automating such scans. There are Indicators of Compromise (IOC) like using obfuscation or PHP functions that are typically used in malware, but that gives false positives even on clean WordPress installation.
Conclusions. Using yara, findbot and Loki may yield a lot of false positives and generates a lot of manual work. But when it comes to finding backdoors it is worth the time. I would recommend using the above tools together with classic forensic work such as looking at timestamps, accessed files in webserver logs, post-data and checksums.
Another article from Jonas Lejon, Backdooring WordPress with Phpsploit, demonstrates how easy it is to modify even a known backdoor a little, making it hard to detect with regular antivirus software. It suggests monitoring all changes with open-source tools like OSSEC or Wazuh.
You don't always need tools developed exclusively for detecting backdoors: you can detect backdoor scripts using grep or findstr, as explained in Finding PHP backdoor scripts from Rinet IT.
It might be easier to detect the injection of malicious content or when someone is using it than finding it from all the places it could hide within the code, as a CMS alone can contains hundreds of thousands of lines of code.
When adding a malicious code, the installation script may try and hide the modification by altering modification dates of all the files and installing the actual backdoors somewhere else than to the files you can see from the logs. In that case, you can see from the logs that your site is probably infected, but where that part is.
Whitelisting is a good way to detect modifications on the core files of the CMS and it's plugins, as you can always download a clean installation and compare the checksums. The settings and contents aren't saved to these files, so the checksums should match.
Daniel Cid: PHP Backdoors: Hidden With Clever Use of Extract Function.
Ananda Krishna: How to Find and Remove Website Backdoors
Detect the malicious actions your site does.
Monitor network traffic from your server. It may e.g. download content from elsewhere or send information. Typically your web server shouldn't be making outbound connection unless it sends email to you or downloads updates from known sources. If there's outboud connections that you don't recognize, it's a bad sign.
Detect modified content on your site.
Keep browser developer tools open and see whether the site gets content from suspicious external sites. Or even better, use Content Security Policy with Report URI like Troy Hunt and automate some of this
This should be the main focus, and detection is the last resort, when all prevention fails.
Keep everything up-to-date. Automate updates, if possible.
Use strong passwords. Add multi-factor authentication, if possible.
Use a web application firewall WAF like ModSecurity to prevent exploiting of 0-day vulnerabilities. Make sure it blocks the actions instead of just logging them.
Delimit access to administrative tools on your site. If you take this to the extreme, you could serve static cached versions of the pages publicly, if they don't actually have dynamic content.
Despite the fact we have grown accustomed to multiple vendors selling us stuff which will detect bad things, it is surprisingly difficult to do and limited in its abilities. While malware detection continues to have its place, a better approach, particularly for hosts acting as servers, is to use white-listing - i.e. identify the stuff which is supposed to be there and report any changes.
This is done with intrusion detection systems. The simplest of these maintain a database of file hashes and check for changes. In the case of a single VPS running PHP + webserver you should at least be monitoring:
Of course, when you run a patching cycle or a code deployment you are going to see lots of alerts unless you re-baseline the file hashes after the exercise.
Make sure you have provisioned a mechanism to get alerts off the target ASAP.
There are lots of other things you can do to ensure the security/integrity of the host, but file checking is an easy and effective mechanism.
Conceptually yes, you can do this. You need to use on an Anti-Virus tool to scan the file system for you, like ClamAV.
