I have a CentOS 6.8 system that happens to be running Tripwire (the Open Source version). The system is on an internal network behind a firewall that is not forwarding any incoming connections. I.e. the Centos system can access the Internet but has a non-routable address (in 10/8) and is not directly or indirectly reachable from outside.
It is being prepared to be an firewall and has no open ports in iptables except for ssh running on a non-standard port and allowing only key-based authentication. I've verified this with nmap.
Yesterday I installed some network debug tools:
Jan 15 15:55:55 Installed: compat-readline5-5.2-17.1.el6.x86_64
Jan 15 15:55:56 Installed: socat-1.7.2.3-1.el6.x86_64
Jan 15 15:56:54 Installed: libnetfilter_conntrack-0.0.100-2.el6.x86_64
Jan 15 15:56:54 Installed: iptstate-2.2.2-4.el6.x86_64
Jan 15 16:06:22 Installed: iperf-2.0.5-11.el6.x86_64
and then ran
tripwire --check -I
to update the database. Everything looked fine in the report, the changes flagged matched the stuff installed by the RPMs.
Tripwire ran overnight and flagged a whole bunch of changes, including changes to the Tripwire binaries.
What I assume this means is that one of those tools is compromised and when I ran it (with sudo, of course) it downloaded and deployed a bunch of stuff. The complete list of changes detected by Tripwire is:
"/usr/sbin"
"/usr/sbin/abrt-auto-reporting"
"/usr/sbin/abrt-dbus"
"/usr/sbin/abrt-server"
"/usr/sbin/arpaname"
"/usr/sbin/crda"
"/usr/sbin/ddns-confgen"
"/usr/sbin/dnssec-dsfromkey"
"/usr/sbin/dnssec-keyfromlabel"
"/usr/sbin/dnssec-keygen"
"/usr/sbin/dnssec-revoke"
"/usr/sbin/dnssec-settime"
"/usr/sbin/dnssec-signzone"
"/usr/sbin/genrandom"
"/usr/sbin/hald"
"/usr/sbin/iftop"
"/usr/sbin/iptstate"
"/usr/sbin/isc-hmac-fixup"
"/usr/sbin/latencytop"
"/usr/sbin/makedumpfile"
"/usr/sbin/mtr"
"/usr/sbin/named-checkconf"
"/usr/sbin/named-checkzone"
"/usr/sbin/named-journalprint"
"/usr/sbin/nethogs"
"/usr/sbin/nsec3hash"
"/usr/sbin/nstat"
"/usr/sbin/ntsysv"
"/usr/sbin/oddjobd"
"/usr/sbin/powertop"
"/usr/sbin/regdbdump"
"/usr/sbin/rndc"
"/usr/sbin/rndc-confgen"
"/usr/sbin/rtacct"
"/usr/sbin/setup"
"/usr/sbin/sssd"
"/usr/sbin/unbound-anchor"
"/usr/sbin/siggen"
"/usr/sbin/tripwire"
"/usr/sbin/twadmin"
"/usr/sbin/twprint"
"/usr/lib/cups/filter"
"/usr/lib/cups/filter/pdftoraster"
"/usr/lib/rpm"
"/usr/lib/rpm/debugedit"
"/usr/lib/rpm/rpmdeps"
"/usr/lib64"
"/usr/lib64/gettext"
"/usr/lib64/gettext/gnu.gettext.DumpResource"
"/usr/lib64/gettext/gnu.gettext.GetURL"
"/usr/lib64/graphviz"
"/usr/lib64/graphviz/libgvplugin_neato_layout.so.6.0.0"
"/usr/lib64/libQt3Support.so.4.6.2"
"/usr/lib64/libQtCLucene.so.4.6.2"
"/usr/lib64/libQtCore.so.4.6.2"
"/usr/lib64/libQtDBus.so.4.6.2"
"/usr/lib64/libQtGui.so.4.6.2"
"/usr/lib64/libQtHelp.so.4.6.2"
"/usr/lib64/libQtNetwork.so.4.6.2"
"/usr/lib64/libQtSql.so.4.6.2"
"/usr/lib64/libQtXml.so.4.6.2"
"/usr/lib64/libSDL-1.2.so.0.11.3"
"/usr/lib64/libXfont.so.1.4.1"
"/usr/lib64/libabrt.so.0.0.1"
"/usr/lib64/libabrt_dbus.so.0.0.1"
"/usr/lib64/libabrt_web.so.0.0.1"
"/usr/lib64/libaugeas.so.0.16.0"
"/usr/lib64/libbind9.so.80.0.4"
"/usr/lib64/libcairo.so.2.10800.8"
"/usr/lib64/libcloog.so.0.0.0"
"/usr/lib64/libcups.so.2"
"/usr/lib64/libcupsimage.so.2"
"/usr/lib64/libdns.so.81.4.1"
"/usr/lib64/libexslt.so.0.8.15"
"/usr/lib64/libfprint.so.0.0.0"
"/usr/lib64/libgd.so.2.0.0"
"/usr/lib64/libgdk-x11-2.0.so.0.2400.23"
"/usr/lib64/libgdk_pixbuf-2.0.so.0.2400.1"
"/usr/lib64/libgmpxx.so.4.1.0"
"/usr/lib64/libgs.so.8.70"
"/usr/lib64/libgstbase-0.10.so.0.25.0"
"/usr/lib64/libgstinterfaces-0.10.so.0.20.0"
"/usr/lib64/libgstreamer-0.10.so.0.25.0"
"/usr/lib64/libgstvideo-0.10.so.0.20.0"
"/usr/lib64/libgtk-x11-2.0.so.0.2400.23"
"/usr/lib64/libgvc.so.5.0.0"
"/usr/lib64/libgvpr.so.1.0.0"
"/usr/lib64/libhunspell-1.2.so.0.0.0"
"/usr/lib64/libisc.so.83.0.3"
"/usr/lib64/libisccc.so.80.0.0"
"/usr/lib64/libisccfg.so.82.0.1"
"/usr/lib64/libjasper.so.1.0.0"
"/usr/lib64/liblcms.so.1.0.19"
"/usr/lib64/liblua-5.1.so"
"/usr/lib64/liblwres.so.80.0.2"
"/usr/lib64/libmng.so.1.0.0"
"/usr/lib64/libnetfilter_conntrack.so.3.0.0"
"/usr/lib64/libnewt.so.0.52.11"
"/usr/lib64/libnfnetlink.so.0.2.0"
"/usr/lib64/libnl-3.so.200.16.1"
"/usr/lib64/libnl-route-3.so.200.16.1"
"/usr/lib64/libopenjpeg.so.2.1.3.0"
"/usr/lib64/libpango-1.0.so.0.2800.1"
"/usr/lib64/libpangocairo-1.0.so.0.2800.1"
"/usr/lib64/libpangoft2-1.0.so.0.2800.1"
"/usr/lib64/libpangox-1.0.so.0.2800.1"
"/usr/lib64/libpangoxft-1.0.so.0.2800.1"
"/usr/lib64/libpathplan.so.4.0.0"
"/usr/lib64/libphonon.so.4.3.1"
"/usr/lib64/libpixman-1.so.0.32.8"
"/usr/lib64/libpng12.so.0.49.0"
"/usr/lib64/libpoppler.so.5.0.0"
"/usr/lib64/libppl.so.7.1.0"
"/usr/lib64/libppl_c.so.2.1.0"
"/usr/lib64/libproxy.so.0.0.0"
"/usr/lib64/libpython2.6.so.1.0"
"/usr/lib64/libreport.so.0.0.1"
"/usr/lib64/librpm.so.1.0.0"
"/usr/lib64/librpmbuild.so.1.0.0"
"/usr/lib64/librpmio.so.1.0.0"
"/usr/lib64/librrd.so.4.0.7"
"/usr/lib64/libsatyr.so.3.0.0"
"/usr/lib64/libslang.so.2.2.1"
"/usr/lib64/libsnappy.so.1.1.4"
"/usr/lib64/libtiff.so.3.9.4"
"/usr/lib64/libunbound.so.2.1.5"
"/usr/lib64/libvorbis.so.0.4.3"
"/usr/lib64/libvorbisenc.so.2.0.6"
"/usr/lib64/libxml2.so.2.7.6"
"/usr/lib64/libxmlrpc.so.3.16"
"/usr/lib64/libxmlrpc_client.so.3.16"
"/usr/lib64/libxslt.so.1.1.26"
"/usr/lib64/perl5/CORE"
"/usr/lib64/perl5/CORE/libperl.so"
"/usr/lib64/qt-3.3/lib"
"/usr/lib64/qt-3.3/lib/libqt-mt.so.3.3.8"
"/sbin"
"/sbin/alsactl"
"/sbin/cryptsetup"
"/sbin/dm_dso_reg_tool"
"/sbin/dmevent_tool"
"/sbin/dmraid"
"/sbin/ifrename"
"/sbin/ip6tables-multi-1.4.7"
"/sbin/iptables-multi-1.4.7"
"/sbin/iw"
"/sbin/iwconfig"
"/sbin/iwevent"
"/sbin/iwgetid"
"/sbin/iwlist"
"/sbin/iwpriv"
"/sbin/iwspy"
"/sbin/kpartx"
"/sbin/parted"
"/sbin/partprobe"
"/sbin/pdata_tools"
"/sbin/plymouthd"
"/sbin/tc"
"/etc"
"/bin"
"/bin/find"
"/bin/gawk"
"/bin/plymouth"
"/bin/rpm"
"/bin/traceroute"
"/lib64"
"/lib64/libasound.so.2.0.0"
"/lib64/libcryptsetup.so.1.1.0"
"/lib64/libdevmapper-event.so.1.02"
"/lib64/libdevmapper.so.1.02"
"/lib64/libdmraid.so.1.0.0.rc16"
"/lib64/libiw.so.29"
"/lib64/libnl.so.1.1.4"
"/lib64/libparted-2.1.so.0.0.0"
"/lib64/libply-splash-core.so.2.0.0"
"/lib64/libply.so.2.0.0"
"/lib64/libreadline.so.5.2"
Most of the changes flagged by Tripwire affect the inode number, MD5 and CRD32, and for some files the change time. A Google search did not turn up any news of a compromise in any of these tools.
Additional Info: Examining /var/log/secure I see that the only sudo command I issued after updating the Tripwire database was iftop, so that might narrow things down a bit.
My first thought was that somehow auto-updates got turned on. However, I can find no evidence that an update happened. There nothing in yum.log and the cron log shows no updates either. Here's a list of currently enabled services:
> chkconfig --list|grep :on
abrt-ccpp 0:off 1:off 2:off 3:on 4:off 5:on 6:off
abrt-oops 0:off 1:off 2:off 3:on 4:off 5:on 6:off
abrtd 0:off 1:off 2:off 3:on 4:off 5:on 6:off
acpid 0:off 1:off 2:on 3:on 4:on 5:on 6:off
atd 0:off 1:off 2:off 3:on 4:on 5:on 6:off
auditd 0:off 1:off 2:on 3:on 4:on 5:on 6:off
autofs 0:off 1:off 2:off 3:on 4:on 5:on 6:off
blk-availability 0:off 1:on 2:on 3:on 4:on 5:on 6:off
cpuspeed 0:off 1:on 2:on 3:on 4:on 5:on 6:off
crond 0:off 1:off 2:on 3:on 4:on 5:on 6:off
dhcpd 0:off 1:off 2:off 3:on 4:on 5:on 6:off
dkms_autoinstaller 0:off 1:off 2:on 3:on 4:on 5:on 6:off
haldaemon 0:off 1:off 2:off 3:on 4:on 5:on 6:off
ip6tables 0:off 1:off 2:on 3:on 4:on 5:on 6:off
iptables 0:off 1:off 2:on 3:on 4:on 5:on 6:off
irqbalance 0:off 1:off 2:off 3:on 4:on 5:on 6:off
kdump 0:off 1:off 2:off 3:on 4:on 5:on 6:off
lvm2-monitor 0:off 1:on 2:on 3:on 4:on 5:on 6:off
mcelogd 0:off 1:off 2:off 3:on 4:off 5:on 6:off
mdmonitor 0:off 1:off 2:on 3:on 4:on 5:on 6:off
messagebus 0:off 1:off 2:on 3:on 4:on 5:on 6:off
named 0:off 1:off 2:off 3:on 4:on 5:on 6:off
netfs 0:off 1:off 2:off 3:on 4:on 5:on 6:off
network 0:off 1:off 2:on 3:on 4:on 5:on 6:off
nmb 0:off 1:off 2:off 3:on 4:on 5:on 6:off
ntpd 0:off 1:off 2:off 3:on 4:on 5:on 6:off
postfix 0:off 1:off 2:on 3:on 4:on 5:on 6:off
rsyslog 0:off 1:off 2:on 3:on 4:on 5:on 6:off
smb 0:off 1:off 2:off 3:on 4:on 5:on 6:off
sshd 0:off 1:off 2:on 3:on 4:on 5:on 6:off
sysstat 0:off 1:on 2:on 3:on 4:on 5:on 6:off
udev-post 0:off 1:on 2:on 3:on 4:on 5:on 6:off
The only "anomaly" here is dkms_autoinstaller which was turned on by a recent update to dkms. However, that appears to run only at boot time and the system hasn't been rebooted in several days.
Here's an extract from the system log, with a few annotations (annotations refer to previous line):
> sudo egrep -v 'dhclient|dhcpd|named' messages
Jan 15 04:21:01 perseus rsyslogd: [origin software="rsyslogd" swVersion="5.8.10" x-pid="1669" x-info="http://www.rsyslog.com"] rsyslogd was HUPed
Jan 15 04:25:12 perseus tripwire[32032]: Integrity Check Complete: /var/lib/tripwire/perseus.jhmg.pvt.twd TWReport perseus.jhmg.pvt 20170115042107 V:4 S:100 A:0 R:0 C:4
^^^ Overnight Tripwire run, 4 violations that I expected
Jan 15 15:53:38 perseus kernel: device eth1 entered promiscuous mode
Jan 15 15:54:03 perseus kernel: device eth1 left promiscuous mode
^^^ Me running tcpdump
Jan 15 15:55:55 perseus yum[2260]: Installed: compat-readline5-5.2-17.1.el6.x86_64
Jan 15 15:55:56 perseus yum[2260]: Installed: socat-1.7.2.3-1.el6.x86_64
Jan 15 15:56:54 perseus yum[2268]: Installed: libnetfilter_conntrack-0.0.100-2.el6.x86_64
Jan 15 15:56:54 perseus yum[2268]: Installed: iptstate-2.2.2-4.el6.x86_64
Jan 15 15:56:59 perseus kernel: Netfilter messages via NETLINK v0.30.
Jan 15 15:56:59 perseus kernel: ctnetlink v0.93: registering with nfnetlink.
Jan 15 16:06:22 perseus yum[2340]: Installed: iperf-2.0.5-11.el6.x86_64
^^^ Installed and tested some network tools
Jan 15 16:12:58 perseus tripwire[2372]: Integrity Check Complete: /var/lib/tripwire/perseus.jhmg.pvt.twd TWReport perseus.jhmg.pvt 20170115160857 V:17 S:100 A:7 R:0 C:10
^^^ Manual execution of Tripwire, database updated
Jan 16 03:29:31 perseus tripwire[6067]: Integrity Check Complete: /var/lib/tripwire/perseus.jhmg.pvt.twd TWReport perseus.jhmg.pvt 20170116032536 V:177 S:100 A:0 R:0 C:177
^^^ Auto run of Tripwire overnight showing 177 violations, all "changes"
