Information Security Stack Exchange is a question and answer site for information security professionals. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have a Django web application where the web app and the postgresql database are hosted on two separate Ubuntu-based machines.

On my Database machine (and not my web application), if I run sudo netstat -4plunt, I get an output as follows:

Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 *.*.*.*:16001           0.0.0.0:*               LISTEN      831/python      
tcp        0      0 127.0.0.1:29131         0.0.0.0:*               LISTEN      46329/mdsd      
tcp        0      0 0.0.0.0:1270            0.0.0.0:*               LISTEN      1582/omiserver  
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1160/sshd       
tcp        0      0 0.0.0.0:5432            0.0.0.0:*               LISTEN      55903/postgres  
udp        0      0 0.0.0.0:51802           0.0.0.0:*                           63911/facebook  
udp        0      0 0.0.0.0:35637           0.0.0.0:*                           63911/facebook  
udp        0      0 0.0.0.0:36013           0.0.0.0:*                           63911/facebook  
udp        0      0 0.0.0.0:52523           0.0.0.0:*                           63911/facebook  
udp        0      0 0.0.0.0:36139           0.0.0.0:*                           63911/facebook  
udp        0      0 0.0.0.0:52618           0.0.0.0:*                           63911/facebook  
udp        0      0 0.0.0.0:36340           0.0.0.0:*                           63911/facebook  
udp        0      0 0.0.0.0:53408           0.0.0.0:*                           63911/facebook  
udp        0      0 0.0.0.0:53711           0.0.0.0:*                           63911/facebook  
udp        0      0 0.0.0.0:53774           0.0.0.0:*                           63911/facebook  
udp        0      0 0.0.0.0:53899           0.0.0.0:*                           63911/facebook  
udp        0      0 0.0.0.0:54031           0.0.0.0:*                           63911/facebook  
udp        0      0 0.0.0.0:54054           0.0.0.0:*                           63911/facebook  
udp        0      0 0.0.0.0:54275           0.0.0.0:*                           63911/facebook  
udp        0      0 0.0.0.0:54375           0.0.0.0:*                           63911/facebook  
udp        0      0 0.0.0.0:38157           0.0.0.0:*                           63911/facebook  
udp        0      0 0.0.0.0:38468           0.0.0.0:*                           63911/facebook  
udp        0      0 0.0.0.0:55391           0.0.0.0:*                           63911/facebook  
udp        0      0 0.0.0.0:39133           0.0.0.0:*                           63911/facebook  
udp        0      0 0.0.0.0:55708           0.0.0.0:*                           63911/facebook  
udp        0      0 0.0.0.0:39963           0.0.0.0:*                           63911/facebook  
udp        0      0 0.0.0.0:40181           0.0.0.0:*                           63911/facebook  
udp        0      0 0.0.0.0:56780           0.0.0.0:*                           63911/facebook  
udp        0      0 0.0.0.0:40441           0.0.0.0:*                           63911/facebook  
udp        0      0 0.0.0.0:56992           0.0.0.0:*                           63911/facebook  

If I run sudo strace -p 63911, I get the following:

Process 63911 attached
[ Process PID=63911 runs in 32 bit mode. ]
select(17, [16], NULL, NULL, {15, 76647}) = 1 (in [16], left {14, 911022})
recv(16, "1", 4096, 0)                  = 1
select(17, [16], NULL, NULL, {20, 0})   = 1 (in [16], left {14, 953682})
recv(16, "1", 4096, 0)                  = 1
select(17, [16], NULL, NULL, {20, 0})   = 1 (in [16], left {14, 910776})
recv(16, "1", 4096, 0)   

This looks faintly like postgresql queries, but I can't tell exactly.

If I run ps -p 63911 -o pid,vsz=MEMORY -o user,group=GROUP -o comm,args=ARGS, I get:

  PID MEMORY USER     GROUP    COMMAND         ARGS
63911 3742124 postgres postgres facebook       /tmp/facebook

And in /tmp/ I see two files. One is an executable called facebook (size: 1.3MB), while the other is called gameover.so (size: 8KB).

I next ran chkrootkit on my machine. I got one positive result: Searching for Suckit rootkit... Warning: /sbin/init INFECTED. However, this is a widely known false positive with chkrootkit. One can't be sure, so I'm going to rebuild the system.

Yet, I don't want to do that without identifying my current vulnerability and improving accordingly. So can anyone give me a heads up of what gameover.so is, and what I can do to protect against these attacks in the future? My firewall might be weak (how do I block all unneeded ports), and as yet I don't have rkhunter (or something similar like Tripwire) installed.


Btw I use the iptables-persistent package, and /etc/iptables/rules.v4 contains the following:

*filter
# Allow all outgoing, but drop incoming and forwarding packets by default
:INPUT DROP [0:0]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [0:0]

# Custom per-protocol chains
:UDP - [0:0]
:TCP - [0:0]
:ICMP - [0:0]

# Acceptable UDP traffic

# Acceptable TCP traffic
-A TCP -p tcp --dport 22 -j ACCEPT
-A TCP -p tcp --dport 5432 -i eth0 -j ACCEPT

# Acceptable ICMP traffic

# Boilerplate acceptance policy
-A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
-A INPUT -i lo -j ACCEPT

# Drop invalid packets
-A INPUT -m conntrack --ctstate INVALID -j DROP

# Pass traffic to protocol-specific chains
## Only allow new connections (established and related should already be handled)
## For TCP, additionally only allow new SYN packets since that is the only valid
## method for establishing a new TCP connection
-A INPUT -p udp -m conntrack --ctstate NEW -j UDP
-A INPUT -p tcp --syn -m conntrack --ctstate NEW -j TCP
-A INPUT -p icmp -m conntrack --ctstate NEW -j ICMP

# Reject anything that's fallen through to this point
## Try to be protocol-specific w/ rejection message
-A INPUT -p udp -j REJECT --reject-with icmp-port-unreachable
-A INPUT -p tcp -j REJECT --reject-with tcp-reset
-A INPUT -j REJECT --reject-with icmp-proto-unreachable

# Commit the changes
COMMIT

*raw
:PREROUTING ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
COMMIT

*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
COMMIT

*security
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
COMMIT

*mangle
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
COMMIT

Running strings /tmp/gameover.so | less yields:

__gmon_start__
_init
_fini
_ITM_deregisterTMCloneTable
_ITM_registerTMCloneTable
__cxa_finalize
_Jv_RegisterClasses
Pg_magic_func
text_ptr_to_char_ptr
malloc
chr_ptr_to_text_ptr
pg_finfo_sys_exec
pg_detoast_datum
system
pfree
pg_finfo_sys_eval
popen
realloc
strncpy
fgets
pclose
pg_finfo_sys_bineval
fork
sysconf
mmap
waitpid
pg_finfo_sys_fileread
fopen
fseek
ftell
fclose
fread
libc.so.6
_edata
__bss_start
_end
GLIBC_2.2.5
fffff.
[]A\
AUATUH
H;] t
Y[]A\D
AWAVAUATUSAQH
<(Ic
[]A\A]A^A_
ATUSH
[]A\
AUATUSH
 []A\A]A^
0123456789ABCDEF
;*3$"
GCC: (Debian 4.7.2-5) 4.7.2
.shstrtab
.note.gnu.build-id
.gnu.hash
.dynsym
.dynstr
.gnu.version
.gnu.version_r
.rela.dyn
.rela.plt
.init
.text
.fini
.rodata
.eh_frame_hdr
.eh_frame
.init_array
.fini_array
.jcr
.dynamic
.got
.got.plt
.data
.bss
.comment

Running strings /tmp/facebook | less yields nomenclature I don't understand:

PTRhP
QVh0
[^_]
<;t 9u
,[^_]
,[^_]
[^_]
[^_]
[^_]
[^_]
[^_]
[^_]
[^_]
[^_]
[^_]
[^_]
\[^_]
[^_]
WVS1
[^_]
[^_]
[^_]
[^_]
        [^_]
[^_]
[^_]
[^_]
[^_]
P[^]
[^_]
XXXX
[^_]
[^_]
[^_]
[^_]
[^_]
[^_]
[^_]
[^_]
[^_]
0[^]
amp.
[^_]
[^_]
[^_]
[^_]
[^_]
[^_]
ffffff.
9Qhu
9Qhu
ffff.
fff.
B`@c
Bd@c
[^_]
,[^_]
[^_]
[^_]
[^_]
CdHc
[^_]
share|improve this question
3  
Related (see also the questions linked from there): How do I deal with a compromised server? on Server Fault. – Michael Kjörling Feb 24 at 14:26
    
@MichaelKjörling: I've been reading those already. But I'm also looking for specifics - how should I change my firewall to account for the open UDP ports that are being exploited, and is gameover.so something others have experienced (I know superficially about the windows-based gameover malware). – Sarah Micj Feb 24 at 14:35
1  
Can you run a query against your entire postgres db for any references for 'facebook' or '/tmp' '/tmp/facebook' 'gameover.so'? – Morgoroth Feb 24 at 19:50
    
@Morgoroth: I searched all tables that contain user inputs. I couldn't find any of the terms you flagged. Maybe should expand my search? – Sarah Micj Feb 24 at 23:14
up vote 6 down vote accepted

Edited:

You seem to be victim of a SQL injection. There is a script (PHP probably) receiving data from web and querying database without enough sanitizing input.

Before you kill 63911, have a look at ps ho lstart 63911 !!

This will let you search in your web server logs what's happened just before process 63911 was run!

First post

  1. What's /tmp/facebook ??

    A) you could strace -p 63911 to have an idea of what this do...

    B) you could tcpdump -ani eth0 udp port 51802 to randomly follow connections.

    C) you could use less /tmp/facebook, or if it's a binary file, strings /tmp/facebook | less (on /tmp/gameover.so too, of course).

  2. What about /sbin/init?

    Build another host with the same version of the same distribution and compare size and date (with ls -l) and sha1sum /sbin/init.

  3. And again: keep your system up to date! And have backups!!

Note: If your system is strongly (and efficiently) infected, you have to shut it down, before doing checksums, string or even ls. But as netstat seems to work without hiding anything, that seems not to be the case...

share|improve this answer
    
Thanks F.Hauri. I ran strace -p 63911 and I got the output [ Process PID=63911 runs in 32 bit mode. ] select(17, [16], NULL, NULL, {19, 501134}) = 1 (in [16], left {14, 922136}) recv(16, "1", 4096, 0) Help in deciphering that? – Sarah Micj Feb 24 at 19:39
    
There is no cipher;-) select wait for an input, recv recieve an input.. and so on. If you find read or write, you may see something between double quotes... – F. Hauri Feb 24 at 19:44
    
Well, postgresql is clearly the way of input of this kit! – F. Hauri Feb 24 at 20:00
    
Answer edited!! – F. Hauri Feb 24 at 20:07
    
So browse your access.log files, search for ip with a lot of access just before Feb 4 18:04 and upto Feb 4 18:10 but few or no more after... So you could see which script was accessed and how. – F. Hauri Feb 24 at 20:14

So can anyone give me a heads up of what has happened

You got hacked. But it can just be anything (SQL injection, remote file inclusion, Shellshock...) and it is impossible to tell without having a close look at the system. And even then it might be impossible to tell because the attacker might have removed any traces how the system was attacked.

Btw I use the iptables-persistent package...

A packet filter firewall like iptables is not able to protect you against application level attacks like SQL injection or similar.

One is an executable called facebook, while the other is called gameover.so (a dynamically linked library?).

Names of files do not matter much, code matters.

...and what I can do to protect against this?

Find out how you got hacked. Fix the problems. Rebuild the server. Don't just restore the server from backup because then you will be probably hacked shortly again with the same attack.

If you are unable to do it yourself get professional help.

share|improve this answer
4  
Given that /tmp/facebook process is being run by the user postgres I'd start investigating SQL injection as the mostly likely attack vector. – Ross Ridge Feb 24 at 17:24
2  
@RossRidge, I thought using Django's ORM (which is what I do) guarded against most SQL injection type attacks, which is why I was discounting this method of attack. Any way to specifically detect whether it was a SQL injection? – Sarah Micj Feb 24 at 17:34
6  
@SarahMicj Your firewall appears to allow anybody to talk directly to PostgreSQL, without visiting your web application. – Michael Hampton Feb 24 at 22:51
    
@MichaelHampton: That's critical, and can be an answer in its own right. Can you expand on that and the remedy? – Sarah Micj Feb 24 at 23:11

This answer addresses, in particular:

what I can do to protect against these attacks in the future

as identifying the actual original infection vector is very difficult, and if there were multiple vulnerabilities - which is likely - insufficient in and of itself. Start with general hardening of the entire set of systems, and add in logging (IDS/IPS) that both can catch infections as they happen, and can block them while they're being attempted.

To really improve, block both directions and start running an IDS/IPS; don't forget to look at those logs, which will be able to help identify the NEXT infection vector.

I recommend you do the following:

  • Unplug your firewall from the Internet in addition to your machines:

    • Buy a new one

    • Or give it a hard reset, and use a non-compromised machine on another network to download any firmware updates for it. If there aren't any because it's old... see "Buy a new one"

      • If they were able to use a computer inside your LAN, they were able to try to, or succeed at, logging into your firewall.

      • Also check to see if your firewall has one of many vulnerabilities; if so, see "Buy a new one"

      • Buy a serious hardware router/firewall, like running pfSense free software on an old machine with 2 NICs, or their own devices, or a fitlet tiny fanless PC, or whatever.

        • Then install the Snort or Suricata package - those are IDS/IPS packages which have a chance of noticing attacks in progress and optionally blocking IPs for a period of time - in your case, I'd say block for a few days and watch your logs very carefully for them to try and get back in.

        • And buy a subscription to the Snort VRT rules; consider the ET Pro rules as well, though they're more expensive.

        • Block EVERYTHING; you'll have massive blocking at first; whitelist on a per-IP*rule basis, and watch them keep trying to get back in.

      • Or at least a Ubiquiti Edgerouter Lite - much cheaper, but very limited in the GUI (all the really advanced stuff is in the command line, and it's definitely not for running an IDS/IPS).

      • And set it up to block EVERYTHING, both inbound and outbound. Open up individual outbound ports on individual IPs or small subnets/blocks as required (for OS updates, for example).

        • This WILL be very annoying - make use of aliases in pfSense, if you go that route

        • This will also seriously restrict the ability of anything inside the firewall to get out to command and control servers.

    • If possible, put the web server and the database server on entirely different VLAN's or firewall ports, so they can only talk to each other via the database connection, one-way, with encryption enabled.

  • Make CERTAIN all your software's fully up to date, and kept there

  • Seek out hardening guides for every piece of software in the mix.

    • And follow them.
  • Change all your passwords on the firewall, servers, database, routers, etc.

    • Use something like KeePass to generate long random ones for each service.

    • Don't forget to go to Database Settings, Security, hit the "1 second delay" link or button, and then increase it - taking 3 or 4 or 7 seconds to open KeePass is a small price to pay for making attackers have to work incredibly much harder than the defaults.

  • Keep the machines off while you:

    • Buy a new hard drive

    • Plug in power but NOT networking (and especially not Wifi; physically remove/turn off all wifi)

    • Install a fresh OS from scratch

    • Set up your software and OS/app level security, and patch it.

    • If the drives in your compromised machines are solid state, take them apart and crack the chips, or apply a torch to them until they melt.

      • Or perform another physical destruction method that protects whatever important data you have left on there, and prevents the drive from being used again.
    • If the drives in your compromised machines have spinning platters, purchase a sledgehammer, a box of gallon Ziplock bags, and a small kitchen towel.

      • Wrap the compromised hard drives in the small kitchen towel

      • Put the wrapped HD in at least 6 Ziplock bags, each sealed

      • Smash the drive with the sledgehammer until glass shards are leaking through the towel, OR it's got significant dents in it (in the case you have metal platters).

      • Restore your valuable data from existing backups

  • Or, much riskier if you don't replaced the compromised HD, wipe it with DBAN or similar and start over

    • If you have any Windows machines on the same network, do several offline virus scans.

      • Use several because no one product covers everything, but by using several different products, you reduce the uncovered space significantly. I'd recommend:

      • at least one of AVG and Avira (or both)

      • at least one of Dr. Web and Kapersky (preferably both), to get some Russian involvement.

      • Comodo Rescue Disk (it advertises rootkit scanning, too)

      • Pick another couple of your favorites.

      • PCSupport.about.com has a pretty good top 15 list as of Feb 2016

      • If you're truly worried, do this on every machine on the network.

        • It won't really take much more time - you can put AVG in one, Avira in another, Kapersky in a third, Dr.Web in a fourth, and then simply move them to the machine on the right when they're done in round-robin fashion.
      • They're almost certainly going to find tracking cookies - that's normal, and nothing to be concerned about, though I'd always delete them.

After all of this, once you're fully online again

  • Make sure your OS is patched, and stays patched.

  • Read your Snort/Suricata logs regularly, and tune the settings.

  • pfSense and the Edgerouter Lite can both set up VPNs of a variety of types. Use a certificate based one, regardless, for any inbound access other than to your webserver

  • Make sure your (new) firewall is patched and up to date.

  • Make sure all other software is patched and up to date.

  • Forever.

share|improve this answer
5  
This reply is not only vague and overkill, it has blatantly wrong information on it. Example: "Smash the drive with the sledgehammer until glass shards are leaking through the towel" ... you could be hammering a long time since most drives don't contain any glass. – Kaithar Feb 24 at 18:15
2  
@Kaithar - I don't know what kind of budget you work with, but most servers I see still use spinning disks. Answer edited to address SSDs as well, however. – Anti-weakpasswords Feb 24 at 18:25
    
@Kaither, I agree that some of this is probably overkill, especially blocking everything (depending on the business model), but the post does mention some useful things. Patching and firewall updates, as well as a good IDS / IPS, and rebuilding are all great advice. – Jonathan Feb 24 at 19:33
4  
@Jonathan it's good advice but not an actual answer to the question which was specifically asking about identifying the infection and attack vector, not highly general opsec tips. Assuming that every drive by scriptkiddie requires a brand new hard drive is going to get expensive fast. Installing without network isn't even going to work if you're doing an net-inst linux setup. – Kaithar Feb 24 at 19:40
1  
Edited to add the portion of the OP that this answer addresses, since without serious forensics, finding out the actual original infection vector is likely to be difficult, and if there were multiple vulnerabilities, insufficient in and of itself. Added an alternate for metal platters, as all mine have been glass for quite awhile. – Anti-weakpasswords Feb 24 at 20:19

Your system was hacked and it looks like your server is being used in a botnet. If you know about when it happened and you have detailed log files then you can analyse the logs to see whether there's an indication of the compromise.

Most systems get hacked by attacking well-known vulnerabilities which have patches available, or bad code that allows attackers to crack your system. The solution is to keep up with your OS and application patching, harden your system and applications, and do a code analysis on your applications to plug any gaps.

share|improve this answer
    
Should the code analysis be on any applications you have developed (if applicable)? – Jonathan Feb 24 at 19:26

You can try Snort: is a powerful network intrusion detection (IDS) tool

Snort's open source network-based intrusion detection system (NIDS) has the ability to perform real-time traffic analysis and packet logging on Internet Protocol (IP) networks. Snort performs protocol analysis, content searching and matching. These basic services have many purposes including application-aware triggered quality of service, to de-prioritize bulk traffic when latency-sensitive applications are in use.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.