Logging server compromised (IptabLes and IptabLex)

Question

I am trying to figure out how it was compromised. They installed IptabLes and IptabLex in /boot.
They also added /etc/init.d/IptabLes and /etc/init.d/IptabLex which simply call the respective /boot files. It seems this attack uses a lot of bandwidth (probably a DDoS); I noticed it immediately.

The server is running CentOS 6.5 with all the latest updates.
It runs logstash, redis, ElasticSearch, and Cherokee webserver serving Kibana.
I am thinking it must either be ElasticSearch or Cherokee web-server.

ElasticSearch port (9200) was open to the world, because Kibana requires it to view the nice graphs. Redis ports (6379) were restricted to only 5 known hosts via iptables.
Cherokee webserver runs on port (8080) not default of 80 and was open to the world.

SSH does not seem to be compromised. The server uses keys and no password authentication
is allowed. We run SSH on port 2020, which is listed as (xinupageserver) in iptables.

Here are the iptables rules. Notice redis is restricted to web hosts,
but http via Cherokee (webcache) and ElasticSearch (wap-wsp) are open.

➜  ~  iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere            
REJECT     all  --  anywhere             loopback/8          reject-with icmp-port-unreachable 
ACCEPT     all  --  anywhere             anywhere            state RELATED,ESTABLISHED 
ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:xinupageserver 
ACCEPT     tcp  --  web1.mydomain.com     anywhere            tcp dpt:6379 
ACCEPT     tcp  --  web2.mydomain.com    anywhere             tcp dpt:6379 
ACCEPT     tcp  --  web3.mydomain.com     anywhere            tcp dpt:6379 
ACCEPT     tcp  --  web4.mydomain.com     anywhere            tcp dpt:6379 
ACCEPT     tcp  --  web5.mydomain.com     anywhere            tcp dpt:6379 
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:wap-wsp 
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:webcache 
ACCEPT     icmp --  anywhere             anywhere            icmp echo-request 
REJECT     all  --  anywhere             anywhere            reject-with icmp-port-unreachable 

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
REJECT     all  --  anywhere             anywhere            reject-with icmp-port-unreachable 

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere 

Thanks so much for the help.

Answer 1

You should never leave Elasticsearch open to the world, in the same way as you wouldn't leave any database server open to the world. Always put a proxy in front of it and use (at least) HTTP auth. Preferably, your proxy should limit what can be done from outside, eg only allowing GET requests, otherwise any user could delete all your data or shutdown your servers.

From version 1.2.0 onwards, dynamic scripting (passing a script as part of eg a search request) has been disabled by default. On older versions it is enabled by default, meaning that any outside user could do anything on your server that the Elasticsearch user can do.

If you're running an older version, you want to add this to your config/elasticsearch.yaml:

script.disable_dynamic: true

See http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-scripting.html#_enabling_dynamic_scripting for more, and http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/url-access-control.html#url-access-control

Answer 2

@Justin I have the same issue. I have monitoring server in DigitalOcean that uses ElasticSearch too.

I think the problem not in ssh because my server uses keys and no password authentication and disabled login for root users.

Some DO users got the same problem with ElasticSearch: https://www.digitalocean.com/community/questions/my-droplet-is-locked-by-support-staff-because-because-of-an-outgoing-flood-or-ddos-what-do-i-do

EIDT: Here is solution how to clean it http://www.ebel-computing.de/JSPWiki/Wiki.jsp?page=VServer%20Trojan but it does not fix the problem in a future

Answer 3

I have had the same issue (IpTablex trojan) on a Debian 7 box with only ssh open via DSL connection. It only ran a postgresql database server which was only listening to internal connections. Highly probable cause was a weak root password, the ssh root login was open. The only other cause could be a compromised windows machine used for logging in to the machine with ssh, but we haven't seen any other machine infected where the windows computers have been used for.

The only good solution is to figure out the cause of infection, backup your important data and reinstall the entire system and fixing the cause of infection. Scripts removing only the /boot/IpTablex and related files is not a solution as it doesn't guarantee there is no other rootkit left somewhere else on the system, and it doesn't fix cause of infection.

Answer 4

Well, immediately, I see a huge flaw in your security posture:

➜  ~  iptables -L Chain INPUT (policy ACCEPT) 
target     prot opt   source    destination
ACCEPT     all  --    anywhere  anywhere

Your first rule on INPUT is accept any inbound connection on any port... so your reject all at the end (and in fact none of your other INPUT rules) matter....

So any of your running/listening services on the server could have been targeted, just FYI.

[EDIT] See proof below

Script started on Sun 27 Jul 2014 11:32:04 AM EDT
]0;nick@fakeit:/tmp[?1034h[nick@fakeit tmp]$ sudo iptables -L
[sudo] password for nick: 
Chain INPUT (policy DROP)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere            
DROP       all  --  anywhere             anywhere            

Chain FORWARD (policy DROP)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
]0;nick@fakeit:/tmp[nick@fakeit tmp]$ ssh iptables_demo@172.16.16.110
The authenticity of host '172.16.16.110 (172.16.16.110)' can't be established.
RSA key fingerprint is fc:86:c7:f6:1a:31:5b:9b:4d:a7:85:ca:7c:f2:29:ec.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '172.16.16.110' (RSA) to the list of known hosts.

iptables_demo@172.16.16.110's password: 
Last login: Sun Jul 27 11:30:55 2014 from localhost

]0;iptables_demo@fakeit:~[?1034h[iptables_demo@fakeit ~]$ echo "I got in with a default INPUT policy of DROP, an explict DROP ALL at the end, but an ACCEPT ALL rule at the beginning"
I got in with a default INPUT policy of DROP, an explict DROP ALL at the end, but an ACCEPT ALL rule at the beginning
]0;iptables_demo@fakeit:~[iptables_demo@fakeit ~]$ exit
logout
Connection to 172.16.16.110 closed.


]0;nick@fakeit:/tmp[nick@fakeit tmp]$ sudo iptables -L
Chain INPUT (policy DROP)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:http 
DROP       all  --  anywhere             anywhere            

Chain FORWARD (policy DROP)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
]0;nick@fakeit:/tmp[nick@fakeit tmp]$ ssh iptables_demo@172.16.16.110
iptables_demo@172.16.16.110's password: 
Last login: Sun Jul 27 11:32:52 2014 from 172.16.16.110

]0;iptables_demo@fakeit:~[?1034h[iptables_demo@fakeit ~]$ echo "I got in with an ACCEPT ALL as the first rule, an ACCEPT HTTP as the second rule, and a DROP ALL as the end rule with a default policy of DROP"
I got in with an ACCEPT ALL as the first rule, an ACCEPT HTTP as the second rule, and a DROP ALL as the end rule with a default policy of DROP
]0;iptables_demo@fakeit:~[iptables_demo@fakeit ~]$ exit
logout
Connection to 172.16.16.110 closed.


]0;nick@fakeit:/tmp[nick@fakeit tmp]$ sudo iptables -L
Chain INPUT (policy DROP)
target     prot opt source               destination         
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:http 
DROP       all  --  anywhere             anywhere            

Chain FORWARD (policy DROP)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
]0;nick@fakeit:/tmp[nick@fakeit tmp]$ ssh iptables_demo@172.16.16.110
ssh: connect to host 172.16.16.110 port 22: Connection timed out

]0;nick@fakeit:/tmp[nick@fakeit tmp]$ echo "I was unable to connect when the only ACCEPT rule was for HTTP and a default DROP policy and an explicit DROP rule"
I was unable to connect when the only ACCEPT rule was for HTTP and a default DROP policy and an explicit DROP rule



]0;nick@fakeit:/tmp[nick@fakeit tmp]$ sudo iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:http 
DROP       all  --  anywhere             anywhere            

Chain FORWARD (policy DROP)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
]0;nick@fakeit:/tmp[nick@fakeit tmp]$ ssh iptables_demo@172.16.16.110
ssh: connect to host 172.16.16.110 port 22: Connection timed out

]0;nick@fakeit:/tmp[nick@fakeit tmp]$ echo "I was unable to connect with the only ACCEPT rule being HTTP and a default ACCEPT policy but an explicit DROP rule at the end of the INPUT filter"
I was unable to connect with the only ACCEPT rule being HTTP and a default ACCEPT policy but an explicit DROP rule at the end of the INPUT filter
]0;nick@fakeit:/tmp[nick@fakeit tmp]$ exit



Script done on Sun 27 Jul 2014 11:38:54 AM EDT