Take the 2-minute tour ×
Information Security Stack Exchange is a question and answer site for Information security professionals. It's 100% free, no registration required.

I have seen some suspicious outbound communications blocked on my VPS, which hosts a small experimental Web site, and want to find out which application is making the attempt. The server is running CentOS and I'm using iptables for the firewall. Here is an excerpt from my iptables rule set:

# create a new chain for rejecting certain ip addresses  
-N input_tcp_deny  
-A input_tcp_deny -m limit --limit 5/min -j LOG --log-prefix "tcp/ip input denied " --log-level 6  
-A input_tcp_deny -j REJECT  

# create a new chain for rejecting certain ip addresses  
-N output_tcp_deny  
-A output_tcp_deny -m limit --limit 5/min -j LOG --log-prefix "tcp/ip output denied " --log-level 4  
-A output_tcp_deny -j REJECT  

-A INPUT -p tcp -s 171/8 -j input_tcp_deny  
-A OUTPUT -p tcp -d 171/8 -j output_tcp_deny  
-A INPUT -p tcp -s 141/8 -j input_tcp_deny  
-A OUTPUT -p tcp -s 141/8 -j output_tcp_deny  
-A INPUT -p tcp -s 103/8 -j input_tcp_deny  
-A OUTPUT -p tcp -d 103/8 -j output_tcp_deny  

iptbles -nvL output shows:

Chain output_tcp_deny (91 references)  
 pkts bytes target     prot opt in     out     source               destination  
    6   120 LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0           limit: avg 5/min burst 5 LOG flags 0 level 4 prefix `tcp/ip output denied '  
    6   120 REJECT     all  --  *      *       0.0.0.0/0            0.0.0.0/0           reject-with icmp-port-unreachable  

Looking in the iptables log file shows:

Apr  9 00:19:25 servername kernel: tcp/ip output denied IN= OUT=eth0 SRC=serverip4addr DST=171.64.65.117 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=15407 DF PROTO=TCP SPT=46898 DPT=80 WINDOW=29200 RES=0x00 SYN URGP=0  
Apr  9 00:19:25 servername kernel: tcp/ip input denied IN=eth0 OUT= MAC=macnotshared SRC=141.214.186.162 DST=serverip4addr LEN=60 TOS=0x00 PREC=0x00 TTL=48 ID=0 DF PROTO=TCP SPT=80 DPT=54582 WINDOW=14480 RES=0x00   ACK SYN URGP=0  
Apr  9 00:19:25 servername kernel: tcp/ip input denied IN=eth0 OUT= MAC=macnotshared SRC=141.219.155.233 DST=serverip4addr LEN=60 TOS=0x00 PREC=0x00 TTL=49 ID=0 DF PROTO=TCP SPT=80 DPT=54760 WINDOW=14480 RES=0x00 ACK SYN URGP=0  
Apr  9 00:19:26 servername kernel: tcp/ip output denied IN= OUT=eth0 SRC=serverip4addr DST=171.64.65.117 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=15408 DF PROTO=TCP SPT=46898 DPT=80 WINDOW=29200 RES=0x00 SYN URGP=0  
Apr  9 00:19:26 servername kernel: tcp/ip input denied IN=eth0 OUT= MAC=macnotshared SRC=141.214.186.162 DST=serverip4addr LEN=60 TOS=0x00 PREC=0x00 TTL=48 ID=0 DF PROTO=TCP SPT=80 DPT=54582 WINDOW=14480 RES=0x00 ACK SYN URGP=0  
Apr  9 00:19:26 servername kernel: tcp/ip input denied IN=eth0 OUT= MAC=macnotshared SRC=141.219.155.233 DST=serverip4addr LEN=60 TOS=0x00 PREC=0x00 TTL=49 ID=0 DF PROTO=TCP SPT=80 DPT=54760 WINDOW=14480 RES=0x00 ACK SYN URGP=0  
Apr  9 00:19:26 servername kernel: tcp/ip input denied IN=eth0 OUT= MAC=macnotshared SRC=141.214.186.162 DST=serverip4addr LEN=60 TOS=0x00 PREC=0x00 TTL=48 ID=0 DF PROTO=TCP SPT=80 DPT=54582 WINDOW=14480 RES=0x00 ACK SYN URGP=0  
Apr 12 11:48:55 servername kernel: tcp/ip input denied IN=eth0 OUT= MAC=macnotshared SRC=141.214.186.162 DST=serverip4addr LEN=60 TOS=0x00 PREC=0x00 TTL=48 ID=0 DF PROTO=TCP SPT=80 DPT=54742 WINDOW=14480 RES=0x00 ACK SYN URGP=0  
Apr 12 11:48:55 servername kernel: tcp/ip input denied IN=eth0 OUT= MAC=macnotshared SRC=141.216.10.182 DST=serverip4addr LEN=60 TOS=0x00 PREC=0x00 TTL=49 ID=0 DF PROTO=TCP SPT=80 DPT=53542 WINDOW=14480 RES=0x00 ACK SYN URGP=0  
Apr 12 11:48:56 servername kernel: tcp/ip input denied IN=eth0 OUT= MAC=macnotshared SRC=141.214.186.162 DST=serverip4addr LEN=60 TOS=0x00 PREC=0x00 TTL=48 ID=0 DF PROTO=TCP SPT=80 DPT=54742 WINDOW=14480 RES=0x00 ACK SYN URGP=0  
Apr 12 11:48:56 servername kernel: tcp/ip input denied IN=eth0 OUT= MAC=macnotshared SRC=141.214.186.162 DST=serverip4addr LEN=60 TOS=0x00 PREC=0x00 TTL=48 ID=0 DF PROTO=TCP SPT=80 DPT=54742 WINDOW=14480 RES=0x00 ACK SYN URGP=0  
Apr 12 11:48:58 servername kernel: tcp/ip output denied IN= OUT=eth0 SRC=serverip4addr DST=103.25.63.44 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=43399 DF PROTO=TCP SPT=45638 DPT=80 WINDOW=29200 RES=0x00 SYN URGP=0  
Apr 12 11:48:58 servername kernel: tcp/ip input denied IN=eth0 OUT= MAC=macnotshared SRC=141.214.186.162 DST=serverip4addr LEN=60 TOS=0x00 PREC=0x00 TTL=48 ID=0 DF PROTO=TCP SPT=80 DPT=54742 WINDOW=14480 RES=0x00 ACK SYN URGP=0  
Apr 12 11:48:59 servername kernel: tcp/ip output denied IN= OUT=eth0 SRC=serverip4addr DST=103.25.63.44 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=43400 DF PROTO=TCP SPT=45638 DPT=80 WINDOW=29200 RES=0x00 SYN URGP=0  
Apr 12 11:49:01 servername kernel: tcp/ip output denied IN= OUT=eth0 SRC=serverip4addr DST=163.178.174.25 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=59335 DF PROTO=TCP SPT=37733 DPT=80 WINDOW=29200 RES=0x00 SYN URGP=0  
Apr 12 11:49:02 servername kernel: tcp/ip output denied IN= OUT=eth0 SRC=serverip4addr DST=163.178.174.25 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=59336 DF PROTO=TCP SPT=37733 DPT=80 WINDOW=29200 RES=0x00 SYN URGP=0

Looking up the IP addresses for blocked outbound communications:
- 171.64.65.117 stanford.edu
- 103.25.63.44 Hong Kong Wholesale Data Center LLC

Inbound blocked communications happened very close in time so I looked those up too:
- 141.214.186.162 University Of Michigan
- 141.219.155.233 mtu.edu (University Of Michigan)
- 141.216.10.182 umflint.edu (University Of Michigan)

For anyone who is curious, I also have IPV6 completely blocked (but no logs) and have seen this from ip6tables -nvL:

Chain OUTPUT (policy DROP 8 packets, 536 bytes)  

So I have dates, times and ip addresses. But now I would like to find out what open source software is doing this.

share|improve this question
1  
I think the answer involves -j ULOG and configuring ulogd. –  Gilles Apr 22 at 11:06

4 Answers 4

If you have auditd installed, you can use it to log connections:

auditctl -a exit,always -F arch=b64 -S connect -k NETWORK

If you have a 32-bit operation system, use b32 rather than b64.

Now you should be able to look in your auditd logs files in /var/log/ to find the processes establishing outgoing connections.

Unfortunately, the audit.log is a bit cryptic. Here is an example from my logs after I run "nc 8.8.8.8 53":

type=SYSCALL msg=audit(1401890039.928:1080289): arch=c000003e syscall=42 success=no exit=-115 a0=3 a1=df3310 a2=10 a3=7fffd8db6910 items=0 ppid=7310 pid=7725 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 ses=4294967295 tty=pts5 comm="nc" exe="/bin/nc.openbsd" key="NETWORK"
type=SOCKADDR msg=audit(1401890039.928:1080289): saddr=02000035080808080000000000000000

In the type=SYSCALL line, you can see the process id, the cmd "/bin/nc.openbsd" and information about the uid running the process. In the type=SOCKADDR line, you have the saddr field containing information about the ip/tcp connection:

02000035080808080000000000000000

02 = IP
35 = Port 53
08080808 = 8.8.8.8

If you want to search for the IP address 163.178.174.25, you could do the following in python to get the hex representation of this ip.

$ python
Python 2.7.4 (default, Sep 26 2013, 03:20:26) 
[GCC 4.7.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import struct, socket
>>> "%x" % struct.unpack("<L", socket.inet_aton("163.178.174.25"))[0]
'19aeb2a3'

Then, you can grep the audit.log for occurrences of this hex string to find the offending application.

And finally, when you have finished the debugging, remember to turn off the auditd rule, as it can produce a fair amount of logs.

auditctl -d exit,always -F arch=b64 -S connect -k NETWORK

Happy hacking!

Credit to answers on serverfault:

http://serverfault.com/questions/352259/finding-short-lived-tcp-connections-owner-process

http://serverfault.com/questions/192893/how-i-can-identify-which-process-is-making-udp-traffic-on-linux

share|improve this answer

You could use a loop to continuously call lsof to see all of the open network connections, log it to a file then grep through it for port 80. That would show you the process opening the socket and the user it's running as.

It could go something like:

#!/bin/bash
# monitor.sh
while true; do
    date
    lsof -i -P
    sleep 1
done

To use it:

$ chmod +x monitor.sh
$ sudo ./monitor.sh > connections.log 

To check it:

$ grep ":80 (" connections.log

By including " (" in grep's pattern to match lsof's output, grep should only match on outbound connections to port 80.

I would grep the log every couple of hours and I would also test it first to make sure the log file wouldn't get too big and hog all my disk space.

share|improve this answer
1  
watch is much nicer than while true; do...done –  Marcin Apr 22 at 19:30

As well as the other good answers (auditd in particular), your situation might call for the following:

  1. Remove the --ulimit parameter from iptables, to make sure you're not missing any connection
  2. Redirect the logs to a different logfile or even better to a different log facility
  3. Configure syslog to catch that facility and redirect to a pipe (see this link for an example)
  4. Write a shell script that reads the content of the pipe and upon receiving an event:
    1. runs lsof or netstat -nalp to capture PIDs and offending binaries
    2. launches a tcpdump session to capture the traffic
    3. e-mails you so you can act quickly :)

My bet is on some harmless piece of software such as a popularity contest, a NTP daemon or a software update. Still I'm curious so do keep us updated please!

share|improve this answer

Having assessed the risk of turning my iptables off for a few minutes, these would be my first steps:

  1. stop the iptables process to allow the open-source app to initiate a connection home (again, only if you feel safe doing this)
  2. run netstat -lntp | less. This list should tell you the destination IPs that your box is connecting to. If you find the ones you have listed above, try and look for the Process ID (PID) for that connection.
  3. run ps aux | less. Try and locate the same PID you have found on step 2 above. You might have to do some digging and play around with the options of this command. This page might be useful.
  4. restart iptables
  5. reboot the server (might not be necessary, but sometimes that is what it takes for the open-source to call home again, unless you already know that the call-home is constant)
  6. check the iptables logs again. Next steps would depend on your findings here.

Because you have only asked on how to identify, rather than block this traffic, I shall stop here; hopefully this will help you a bit.

share|improve this answer
    
This will only work if you can catch the connection while it's up, i.e. only if the connection lasts sufficiently long. There's no such thing as “stop the iptables process”: iptables is the program that changes the settings, not a program that filters traffic; you mean switching the firewall to not block these connections, which doesn't require switching it to being wide open. –  Gilles Apr 22 at 11:08

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.