What steps need to do to identify security hole

Question

I use VPS hosting. Installed CentOS 6 and ZPanel according to this tutorial http://www.zvps.co.uk/zpanelcp/centos-6

Received warning from hosting provider: This URL has been identified as a phishing site and is currently involved in identity theft activities. URL: hxxp://111.11.238.177/www.connet-itunes.fr/iTunesConnect.woasp/ (url here and further are a bit modified)

Downloaded server files to local computer. Searched for itunes. Found logs like these

\var\log\httpd\access_log
111.11.160.253 - - [20/Oct/2013:15:18:07 -0400] "GET /www.connet-itunes.fr/iTunesConnect.woasp/ HTTP/1.1" 200 1326 "-" "curl/7.18.2 (i486-pc-linux-gnu) libcurl/7.18.2 OpenSSL/0.9.8g zlib/1.2.3.3 libidn/1.10"
11.111.144.148 - - [20/Oct/2013:15:30:13 -0400] "GET /www.connet-itunes.fr/iTunesConnect.woasp/ HTTP/1.1" 200 1326 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows\t\t\t\t\tNT 5.2)"
11.111.33.36 - - [14/Oct/2013:03:26:49 -0400] "POST /www.connet-itunes.fr/iTunesConnect.woasp/inscription/ HTTP/1.1" 200 13392 "http://176.56.238.177/www.connet-itunes.fr/iTunesConnect.woasp/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_5_8) AppleWebKit/534.50.2 (KHTML, like Gecko) Version/5.0.6 Safari/533.22.3"

\var\zpanel\logs\domains\zadmin\mydomainname.com-error.log 1 443,00 KB 23.10.2013 09:12:56
[Tue Oct 15 02:47:30 2013] [error] [client 11.11.253.220] File does not exist: /var/zpanel/hostdata/zadmin/public_html/mydomainname_com/www.connet-itunes.fr, referer: http://www.mydomainname.com/www.connet-itunes.fr/iTunesConnect.woasp/

\var\var\zpanel\logs\domains\zadmin\mydomainname.com-access.log 111,00 KB 23.10.2013 09:12:56
11.11.253.220 - - [15/Oct/2013:02:47:30 -0400] "GET /www.connet-itunes.fr/iTunesConnect.woasp/ HTTP/1.0" 404 - "http://www.mydomainname.com/www.connet-itunes.fr/iTunesConnect.woasp/" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 2.0.50727)"

\var\zpanel\logs\domains\zadmin\anotherdomain.lv-access.log 4 386,00 KB 23.10.2013 14:02:56
11.11.164.110 - - [15/Oct/2013:02:47:36 -0400] "GET /www.connet-itunes.fr/iTunesConnect.woasp/ HTTP/1.0" 404 42823 "http://www.anotherdomain.lv/www.connet-itunes.fr/iTunesConnect.woasp/" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 2.0.50727)"

\var\zpanel\logs\domains\zadmin\anotherdomain.lv-error.log 196,00 KB 23.10.2013 12:41:40
[Tue Oct 15 02:47:36 2013] [error] [client 11.11.164.110] File does not exist: /var/zpanel/hostdata/zadmin/public_html/anotherdomain_lv/www.connet-itunes.fr, referer: http://www.anotherdomain.lv/www.connet-itunes.fr/iTunesConnect.woasp/

\var\log\httpd\error_log 36,00 KB 23.10.2013 13:08:36
[Tue Oct 15 03:50:03 2013] [error] [client 11.111.23.90] Directory index forbidden by Options directive: /etc/zpanel/panel/www.connet-itunes.fr/
[Tue Oct 15 05:28:28 2013] [error] [client 11.111.140.53] File does not exist: /etc/zpanel/panel/favicon.ico, referer: http://111.11.238.177/www.connet-itunes.fr/iTunesConnect.woasp/
[Sat Oct 19 17:46:38 2013] [error] [client 5.49.165.166] PHP Notice:  Undefined index: donnee2 in /etc/zpanel/panel/www.connet-itunes.fr/iTunesConnect.woasp/inscription/index.php on line 277

Found one file (virustotal.com identified as PHP:Shell-BH [Trj], PHP/BackDoor.BT, VEX16c0.Webshell, PHP/PhpShell.NAJ, Backdoor:PHP/Shell.Q). Using that file hacker seems can get full control by server files (see, delete, upload, modify); I checked the file on local computer (wamp).

But from all the above I have no idea where is security hole. Location of that file was etc/zpanel/panel/bin/. Does it means that hacker hacked ZPanel? Found one related discussion here http://forums.zpanelcp.com/thread-8604.html But no solution. Please advice what steps need to do to identify security holes

Answer 1

Unfortunately, the log entries you show do not necessarily relate to the vulnerability which allowed the attacker to enter in the first place. These logs show that the attacker pushed an infected file on your server and then used your server as an helper host in order to infect other people -- your server was used as a simple file server. The URL beginning with "http://", then your IP address, then a "/", then a "seemingly honest server name", is an obvious attempt at making human users believe that they are downloading the file from an official iTunes source, and not your server.

The fact that the file was located within the ZPanel directories may indicate that the vulnerability used by the attacker did not grant him full access, but only allowed him to push a file of his own, in a directory that the attacker does not fully control. This would be a good case for you: it would mean that the vulnerability is indeed in ZPanel, and that your machine may be salvageable (i.e. it was only partially breached into). Note that a few Google searches reveal that the security of ZPanel is being questioned.

However, we cannot logically exclude a darker scenario, where the attacker took full control of the machine, and pushed the file in the ZPanel directory only so that you would have low probability of stumbling upon it yourself. If that is the actual scenario, then you will have to reformat the complete server (the "nuke for orbit" cure).

In any case, in order to identify the first vulnerability, you have to look for earlier log entries. The dates of creation / modification of the file on your filesystem may give clues as to when the file was pushed on your server. This may have happened quite earlier (the attacker may have left the file "dormant" for some days or weeks before actually using it for his phishing scam).