I was pulled up by my ISP in work for having a machine making dozens of outbound SSH connections. I tracked down the machine and took it off the network. Problem is there are several other machines with identical configs that potentially could suffer a similar fate.

I can see outbound connection attempts using tcpdump but im having trouble trying to track down the script or process making these connection attempts. I've been trying to use netstat -plunt, ss -tp, lsof and various commandline utils but haven't been able to source the offending process. Has anyone any advice on tracking down processes and applications on compromised boxes like this?

sudo netstat -plunt

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0*               LISTEN      1339/redis-server 1 
tcp        0      0  *               LISTEN      1008/systemd-resolv 
tcp        0      0    *               LISTEN      1024/sshd           
tcp        0      0 *               LISTEN      1413/cupsd          
tcp6       0      0 :::5355                 :::*                    LISTEN      1008/systemd-resolv 
tcp6       0      0 :::4949                 :::*                    LISTEN      1022/perl           
tcp6       0      0 :::22                   :::*                    LISTEN      1024/sshd           
tcp6       0      0 ::1:631                 :::*                    LISTEN      1413/cupsd          
udp        0      0 *                           938/avahi-daemon: r 
udp        0      0 *                           1364/openvpn        
udp        0      0  *                           938/avahi-daemon: r 
udp        0      0  *                           1008/systemd-resolv 
udp        0      0 *                           1008/systemd-resolv 
udp        0      0   *                           1414/cups-browsed   
udp6       0      0 :::53487                :::*                                938/avahi-daemon: r 
udp6       0      0 :::5353                 :::*                                938/avahi-daemon: r 
udp6       0      0 :::5355                 :::*                                1008/systemd-resolv 
  • "but haven't been able to source the offending process" - do you see the connections, but not the process, or can you not see the connections at all? – schroeder Jul 2 '18 at 11:12
  • What do you see from netstat? One hacky alternative may be to replace the ssh client on the machine with a script that logs the parent PID / name. – Hector Jul 2 '18 at 12:59
  • @schroeder I don't see the connections. tcpdump shows the traffic. netstat doesn't show these client connections outbound. @Hector I could ln -s to a logging process I suppose. I thought there might be something really basic I'm missing. Some terminal command that I'm not aware of ... tcp 0 0* LISTEN 1024/sshd tcp 0 0* LISTEN 1413/cupsd ... Sample output. I don't see any of the foreign addresses from tcpdump listed on netstat. – dcos Jul 2 '18 at 15:45
  • What may be happening is that the connections are sporadic. You would need to log the changes over time to see the connections. – schroeder Jul 2 '18 at 15:48
  • 1
    netstat -plunt includes the -l flag so it only shows listening connections. Outbound connections won't be shown by it. ssh is tcp so netstat -pnt would probably work or the ss -pt you tried but haven't provided the output for. – William Hay Sep 1 '18 at 9:20

Do you think that you can have a rootkit installed that does not allow you to see the offending process?

Another idea is to use the command watch, to try to spot the offending process.

watch -d 'lsof -i tcp:22'

watch -d highlights the differences. It runs the command every 2 secs.

lsof -i tcp:22 gives you a list of processes that use TCP connections on port 22.

In case you suspect of a rootkit, you can compile busybox and use it to check your processes and open files.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.