www-data is sending rogue mails through sendmail. How do I find the source?

Question

So I have a Ubuntu server hosting a PHP-based web site for me. Some of the services rely on sendmail being configured. I have set it up to send through my GMail account.

As of yesterday ~19:00 CET, my GMail "sent mail" folder suddenly got overwhelmed with Returned mail delivery reports from Mail Delivery Subsystem. I could see that someone is attempting to send outgoing emails (spam) from my system using the www-data. Luckily for me, they were also attempting to alter the From-field, which was denied by (AFAIK) Google. Here is a (sensored) transcript from mail.log:

Feb  4 18:58:10 ip-xxxxx sendmail[740]: s14IwAHQ000740: Authentication-Warning: ip-xxxxx.ec2.internal: www-data set sender to [email protected] using -f
Feb  4 18:58:10 ip-xxxxx sendmail[740]: s14IwAHQ000740: [email protected], size=464, class=0, nrcpts=1, msgid=<[email protected]>, relay=www-data@localhost

I've "shut down" sendmail by chmod'ing the executable to 000.

So the thing is I'd like to get sendmail up and running again while closing the security hole. I'm a bit of a loss of where to start. I'm not a Linux expert although I've managed to set this system up.

I'm running a total of 5 web sites (Virtual hosts) on the system, and I'm pretty confident that one of them, and which one of them, is compromised. In the log above, I've swapped the real domain of one of my 5 sites with "mydomain.com". So I'm pretty sure it's that particular domain which is attacked. I can't find any suspicious activity in the apache access log though. Where do I go from here?

edit1: Anyone got any tips on how I can find out whether or not a user got full control of my www-data account, or if it goes through HTTP calls to a PHP file?

Edit2: Found this nugget of a file when searching for PHP files modified the last 48 hours (I had personally modified zero).

System:

• Ubuntu 12.04 LTS running at Amazon EC2
• PHP version: 5.3.10-1ubuntu3.2 with Suhosin-Patch (cli) (built: Jun 13 2012 17:19:58)
• Apache2 version: 2.2.22
• Website running PHP Fusion CMS system v7.01.01

Answer 1

From your update I'd guess that PHP fusion is the problem here. It looks like there's wide range of vulnerabilites for newer versions than that including SQL Injection and other pretty high risk stuff (sample info here), so it's reasonable to assume that the one you're using is vulnerable too...

So it would be likely that your attackers have got access to your server that way with at least the privileges of the web server user.

At this point really you need to look at rebuilding the server as it's very hard to effectively clean someone off your system without knowing exactly what they've done (e.g. put rootkits on the system)

If you have backups from before the problem started you could work from that, although it could be tricky to tell exactly when they got access.

Key point would be to ensure that you have fully updated versions of all the software you use and make sure to keep the CMS updated as they can be common points of attack.

Also looks like there's some advice on the php fusion pages (here) on recovering from this kind of attack

Answer 2

I managed to trace the files without rebuilding the whole server which are sending spam emails. This is probably happening because of any PHP script that is running on your server. Usually hackers are deploying these scripts that are using eval() methods to execute the php mail code randomly.

I found Files with name such as

• db.php
• functions90.php
• page35.php
• start.pgp
• system31.php
• template.php
• title.php

How did I find these files? I enabled the Debug option for ssmtp by adding one line in ssmtp.config file DEBUG = YES. Then I went to /var/logs/mail.log and I found some content like:

<[email protected]>
Jun 21 10:22:54 megatron sSMTP[19726]: To: [email protected]
Jun 21 10:22:54 megatron sSMTP[19726]: Subject: I've been thinking of you
Jun 21 10:22:54 megatron sSMTP[19726]: X-PHP-Originating-Script: 1002:db.php(1965) : eval()'d code
Jun 21 10:22:54 megatron sSMTP[19726]: Date: Tue, 21 Jun 2016 10:22:54 +0100
Jun 21 10:22:54 megatron sSMTP[19726]: Message-ID: <[email protected]>
Jun 21 10:22:54 megatron sSMTP[19726]: X-Priority: 3
Jun 21 10:22:54 megatron sSMTP[19726]: MIME-Version: 1.0
Jun 21 10:22:54 megatron sSMTP[19726]: Content-Type: multipart/alternative;#015#012#011boundary="b1_a27900aabf9e492a7f0f180afaa7902f"
Jun 21 10:22:54 megatron sSMTP[19726]: Content-Transfer-Encoding: 8bit
Jun 21 10:22:54 megatron sSMTP[19726]: 
Jun 21 10:22:54 megatron sSMTP[19726]: --b1_a27900aabf9e492a7f0f180afaa7902f
Jun 21 10:22:54 megatron sSMTP[19726]: Content-Type: text/plain; charset=us-ascii
Jun 21 10:22:54 megatron sSMTP[19726]: 
Jun 21 10:22:54 megatron sSMTP[19726]: Wanna play with my vibrator in my bedroom?
Jun 21 10:22:54 megatron sSMTP[19726]: 
Jun 21 10:22:54 megatron sSMTP[19726]: Don't break my heart, sweetie!
Jun 21 10:22:54 megatron sSMTP[19726]: 
Jun 21 10:22:54 megatron sSMTP[19726]: Aren't you going to ignore me?
Jun 21 10:22:54 megatron sSMTP[19726]: 

Now if you closely look at the 4th line of the Debug log X-PHP-Originating-Script: it is showing that db.php is running this script using eval()'d code.

Solution:
I noted down all files found in the logs and then find them by using find / -name "*filename*.php" > /var/www/filesSearch.log and delete all files by taking a backup. After deleting the files in the files search log, I rechecked all of my folders' permissions.

You must check all of the www folder permissions again and keep most of them to 755 or 644. If you are uploading any file using website then please don't allow to upload php, sh or server-side executable files.