0

I have a CentOS 6.8 system that happens to be running Tripwire (the Open Source version). The system is on an internal network behind a firewall that is not forwarding any incoming connections. I.e. the Centos system can access the Internet but has a non-routable address (in 10/8) and is not directly or indirectly reachable from outside.

It is being prepared to be an firewall and has no open ports in iptables except for ssh running on a non-standard port and allowing only key-based authentication. I've verified this with nmap.

Yesterday I installed some network debug tools:

Jan 15 15:55:55 Installed: compat-readline5-5.2-17.1.el6.x86_64
Jan 15 15:55:56 Installed: socat-1.7.2.3-1.el6.x86_64
Jan 15 15:56:54 Installed: libnetfilter_conntrack-0.0.100-2.el6.x86_64
Jan 15 15:56:54 Installed: iptstate-2.2.2-4.el6.x86_64
Jan 15 16:06:22 Installed: iperf-2.0.5-11.el6.x86_64

and then ran

tripwire --check -I 

to update the database. Everything looked fine in the report, the changes flagged matched the stuff installed by the RPMs.

Tripwire ran overnight and flagged a whole bunch of changes, including changes to the Tripwire binaries.

What I assume this means is that one of those tools is compromised and when I ran it (with sudo, of course) it downloaded and deployed a bunch of stuff. The complete list of changes detected by Tripwire is:

"/usr/sbin"
"/usr/sbin/abrt-auto-reporting"
"/usr/sbin/abrt-dbus"
"/usr/sbin/abrt-server"
"/usr/sbin/arpaname"
"/usr/sbin/crda"
"/usr/sbin/ddns-confgen"
"/usr/sbin/dnssec-dsfromkey"
"/usr/sbin/dnssec-keyfromlabel"
"/usr/sbin/dnssec-keygen"
"/usr/sbin/dnssec-revoke"
"/usr/sbin/dnssec-settime"
"/usr/sbin/dnssec-signzone"
"/usr/sbin/genrandom"
"/usr/sbin/hald"
"/usr/sbin/iftop"
"/usr/sbin/iptstate"
"/usr/sbin/isc-hmac-fixup"
"/usr/sbin/latencytop"
"/usr/sbin/makedumpfile"
"/usr/sbin/mtr"
"/usr/sbin/named-checkconf"
"/usr/sbin/named-checkzone"
"/usr/sbin/named-journalprint"
"/usr/sbin/nethogs"
"/usr/sbin/nsec3hash"
"/usr/sbin/nstat"
"/usr/sbin/ntsysv"
"/usr/sbin/oddjobd"
"/usr/sbin/powertop"
"/usr/sbin/regdbdump"
"/usr/sbin/rndc"
"/usr/sbin/rndc-confgen"
"/usr/sbin/rtacct"
"/usr/sbin/setup"
"/usr/sbin/sssd"
"/usr/sbin/unbound-anchor"

"/usr/sbin/siggen"
"/usr/sbin/tripwire"
"/usr/sbin/twadmin"
"/usr/sbin/twprint"
"/usr/lib/cups/filter"
"/usr/lib/cups/filter/pdftoraster"
"/usr/lib/rpm"
"/usr/lib/rpm/debugedit"
"/usr/lib/rpm/rpmdeps"

"/usr/lib64"
"/usr/lib64/gettext"
"/usr/lib64/gettext/gnu.gettext.DumpResource"
"/usr/lib64/gettext/gnu.gettext.GetURL"
"/usr/lib64/graphviz"
"/usr/lib64/graphviz/libgvplugin_neato_layout.so.6.0.0"
"/usr/lib64/libQt3Support.so.4.6.2"
"/usr/lib64/libQtCLucene.so.4.6.2"
"/usr/lib64/libQtCore.so.4.6.2"
"/usr/lib64/libQtDBus.so.4.6.2"
"/usr/lib64/libQtGui.so.4.6.2"
"/usr/lib64/libQtHelp.so.4.6.2"
"/usr/lib64/libQtNetwork.so.4.6.2"
"/usr/lib64/libQtSql.so.4.6.2"
"/usr/lib64/libQtXml.so.4.6.2"
"/usr/lib64/libSDL-1.2.so.0.11.3"
"/usr/lib64/libXfont.so.1.4.1"
"/usr/lib64/libabrt.so.0.0.1"
"/usr/lib64/libabrt_dbus.so.0.0.1"
"/usr/lib64/libabrt_web.so.0.0.1"
"/usr/lib64/libaugeas.so.0.16.0"
"/usr/lib64/libbind9.so.80.0.4"
"/usr/lib64/libcairo.so.2.10800.8"
"/usr/lib64/libcloog.so.0.0.0"
"/usr/lib64/libcups.so.2"
"/usr/lib64/libcupsimage.so.2"
"/usr/lib64/libdns.so.81.4.1"
"/usr/lib64/libexslt.so.0.8.15"
"/usr/lib64/libfprint.so.0.0.0"
"/usr/lib64/libgd.so.2.0.0"
"/usr/lib64/libgdk-x11-2.0.so.0.2400.23"
"/usr/lib64/libgdk_pixbuf-2.0.so.0.2400.1"
"/usr/lib64/libgmpxx.so.4.1.0"
"/usr/lib64/libgs.so.8.70"
"/usr/lib64/libgstbase-0.10.so.0.25.0"
"/usr/lib64/libgstinterfaces-0.10.so.0.20.0"
"/usr/lib64/libgstreamer-0.10.so.0.25.0"
"/usr/lib64/libgstvideo-0.10.so.0.20.0"
"/usr/lib64/libgtk-x11-2.0.so.0.2400.23"
"/usr/lib64/libgvc.so.5.0.0"
"/usr/lib64/libgvpr.so.1.0.0"
"/usr/lib64/libhunspell-1.2.so.0.0.0"
"/usr/lib64/libisc.so.83.0.3"
"/usr/lib64/libisccc.so.80.0.0"
"/usr/lib64/libisccfg.so.82.0.1"
"/usr/lib64/libjasper.so.1.0.0"
"/usr/lib64/liblcms.so.1.0.19"
"/usr/lib64/liblua-5.1.so"
"/usr/lib64/liblwres.so.80.0.2"
"/usr/lib64/libmng.so.1.0.0"
"/usr/lib64/libnetfilter_conntrack.so.3.0.0"
"/usr/lib64/libnewt.so.0.52.11"
"/usr/lib64/libnfnetlink.so.0.2.0"
"/usr/lib64/libnl-3.so.200.16.1"
"/usr/lib64/libnl-route-3.so.200.16.1"
"/usr/lib64/libopenjpeg.so.2.1.3.0"
"/usr/lib64/libpango-1.0.so.0.2800.1"
"/usr/lib64/libpangocairo-1.0.so.0.2800.1"
"/usr/lib64/libpangoft2-1.0.so.0.2800.1"
"/usr/lib64/libpangox-1.0.so.0.2800.1"
"/usr/lib64/libpangoxft-1.0.so.0.2800.1"
"/usr/lib64/libpathplan.so.4.0.0"
"/usr/lib64/libphonon.so.4.3.1"
"/usr/lib64/libpixman-1.so.0.32.8"
"/usr/lib64/libpng12.so.0.49.0"
"/usr/lib64/libpoppler.so.5.0.0"
"/usr/lib64/libppl.so.7.1.0"
"/usr/lib64/libppl_c.so.2.1.0"
"/usr/lib64/libproxy.so.0.0.0"
"/usr/lib64/libpython2.6.so.1.0"
"/usr/lib64/libreport.so.0.0.1"
"/usr/lib64/librpm.so.1.0.0"
"/usr/lib64/librpmbuild.so.1.0.0"
"/usr/lib64/librpmio.so.1.0.0"
"/usr/lib64/librrd.so.4.0.7"
"/usr/lib64/libsatyr.so.3.0.0"
"/usr/lib64/libslang.so.2.2.1"
"/usr/lib64/libsnappy.so.1.1.4"
"/usr/lib64/libtiff.so.3.9.4"
"/usr/lib64/libunbound.so.2.1.5"
"/usr/lib64/libvorbis.so.0.4.3"
"/usr/lib64/libvorbisenc.so.2.0.6"
"/usr/lib64/libxml2.so.2.7.6"
"/usr/lib64/libxmlrpc.so.3.16"
"/usr/lib64/libxmlrpc_client.so.3.16"
"/usr/lib64/libxslt.so.1.1.26"
"/usr/lib64/perl5/CORE"
"/usr/lib64/perl5/CORE/libperl.so"
"/usr/lib64/qt-3.3/lib"
"/usr/lib64/qt-3.3/lib/libqt-mt.so.3.3.8"

"/sbin"
"/sbin/alsactl"
"/sbin/cryptsetup"
"/sbin/dm_dso_reg_tool"
"/sbin/dmevent_tool"
"/sbin/dmraid"
"/sbin/ifrename"
"/sbin/ip6tables-multi-1.4.7"
"/sbin/iptables-multi-1.4.7"
"/sbin/iw"
"/sbin/iwconfig"
"/sbin/iwevent"
"/sbin/iwgetid"
"/sbin/iwlist"
"/sbin/iwpriv"
"/sbin/iwspy"
"/sbin/kpartx"
"/sbin/parted"
"/sbin/partprobe"
"/sbin/pdata_tools"
"/sbin/plymouthd"
"/sbin/tc"

"/etc"

"/bin"
"/bin/find"
"/bin/gawk"
"/bin/plymouth"
"/bin/rpm"
"/bin/traceroute"

"/lib64"
"/lib64/libasound.so.2.0.0"
"/lib64/libcryptsetup.so.1.1.0"
"/lib64/libdevmapper-event.so.1.02"
"/lib64/libdevmapper.so.1.02"
"/lib64/libdmraid.so.1.0.0.rc16"
"/lib64/libiw.so.29"
"/lib64/libnl.so.1.1.4"
"/lib64/libparted-2.1.so.0.0.0"
"/lib64/libply-splash-core.so.2.0.0"
"/lib64/libply.so.2.0.0"
"/lib64/libreadline.so.5.2"

Most of the changes flagged by Tripwire affect the inode number, MD5 and CRD32, and for some files the change time. A Google search did not turn up any news of a compromise in any of these tools.

Additional Info: Examining /var/log/secure I see that the only sudo command I issued after updating the Tripwire database was iftop, so that might narrow things down a bit.

My first thought was that somehow auto-updates got turned on. However, I can find no evidence that an update happened. There nothing in yum.log and the cron log shows no updates either. Here's a list of currently enabled services:

> chkconfig --list|grep :on
abrt-ccpp       0:off   1:off   2:off   3:on    4:off   5:on    6:off
abrt-oops       0:off   1:off   2:off   3:on    4:off   5:on    6:off
abrtd           0:off   1:off   2:off   3:on    4:off   5:on    6:off
acpid           0:off   1:off   2:on    3:on    4:on    5:on    6:off
atd             0:off   1:off   2:off   3:on    4:on    5:on    6:off
auditd          0:off   1:off   2:on    3:on    4:on    5:on    6:off
autofs          0:off   1:off   2:off   3:on    4:on    5:on    6:off
blk-availability    0:off   1:on    2:on    3:on    4:on    5:on    6:off
cpuspeed        0:off   1:on    2:on    3:on    4:on    5:on    6:off
crond           0:off   1:off   2:on    3:on    4:on    5:on    6:off
dhcpd           0:off   1:off   2:off   3:on    4:on    5:on    6:off
dkms_autoinstaller  0:off   1:off   2:on    3:on    4:on    5:on    6:off
haldaemon       0:off   1:off   2:off   3:on    4:on    5:on    6:off
ip6tables       0:off   1:off   2:on    3:on    4:on    5:on    6:off
iptables        0:off   1:off   2:on    3:on    4:on    5:on    6:off
irqbalance      0:off   1:off   2:off   3:on    4:on    5:on    6:off
kdump           0:off   1:off   2:off   3:on    4:on    5:on    6:off
lvm2-monitor    0:off   1:on    2:on    3:on    4:on    5:on    6:off
mcelogd         0:off   1:off   2:off   3:on    4:off   5:on    6:off
mdmonitor       0:off   1:off   2:on    3:on    4:on    5:on    6:off
messagebus      0:off   1:off   2:on    3:on    4:on    5:on    6:off
named           0:off   1:off   2:off   3:on    4:on    5:on    6:off
netfs           0:off   1:off   2:off   3:on    4:on    5:on    6:off
network         0:off   1:off   2:on    3:on    4:on    5:on    6:off
nmb             0:off   1:off   2:off   3:on    4:on    5:on    6:off
ntpd            0:off   1:off   2:off   3:on    4:on    5:on    6:off
postfix         0:off   1:off   2:on    3:on    4:on    5:on    6:off
rsyslog         0:off   1:off   2:on    3:on    4:on    5:on    6:off
smb             0:off   1:off   2:off   3:on    4:on    5:on    6:off
sshd            0:off   1:off   2:on    3:on    4:on    5:on    6:off
sysstat         0:off   1:on    2:on    3:on    4:on    5:on    6:off
udev-post       0:off   1:on    2:on    3:on    4:on    5:on    6:off

The only "anomaly" here is dkms_autoinstaller which was turned on by a recent update to dkms. However, that appears to run only at boot time and the system hasn't been rebooted in several days.

Here's an extract from the system log, with a few annotations (annotations refer to previous line):

> sudo egrep -v 'dhclient|dhcpd|named' messages
Jan 15 04:21:01 perseus rsyslogd: [origin software="rsyslogd" swVersion="5.8.10" x-pid="1669" x-info="http://www.rsyslog.com"] rsyslogd was HUPed
Jan 15 04:25:12 perseus tripwire[32032]: Integrity Check Complete: /var/lib/tripwire/perseus.jhmg.pvt.twd TWReport perseus.jhmg.pvt 20170115042107 V:4 S:100 A:0 R:0 C:4
^^^ Overnight Tripwire run, 4 violations that I expected

Jan 15 15:53:38 perseus kernel: device eth1 entered promiscuous mode
Jan 15 15:54:03 perseus kernel: device eth1 left promiscuous mode
^^^ Me running tcpdump

Jan 15 15:55:55 perseus yum[2260]: Installed: compat-readline5-5.2-17.1.el6.x86_64
Jan 15 15:55:56 perseus yum[2260]: Installed: socat-1.7.2.3-1.el6.x86_64
Jan 15 15:56:54 perseus yum[2268]: Installed: libnetfilter_conntrack-0.0.100-2.el6.x86_64
Jan 15 15:56:54 perseus yum[2268]: Installed: iptstate-2.2.2-4.el6.x86_64
Jan 15 15:56:59 perseus kernel: Netfilter messages via NETLINK v0.30.
Jan 15 15:56:59 perseus kernel: ctnetlink v0.93: registering with nfnetlink.
Jan 15 16:06:22 perseus yum[2340]: Installed: iperf-2.0.5-11.el6.x86_64
^^^ Installed and tested some network tools

Jan 15 16:12:58 perseus tripwire[2372]: Integrity Check Complete: /var/lib/tripwire/perseus.jhmg.pvt.twd TWReport perseus.jhmg.pvt 20170115160857 V:17 S:100 A:7 R:0 C:10
^^^ Manual execution of Tripwire, database updated

Jan 16 03:29:31 perseus tripwire[6067]: Integrity Check Complete: /var/lib/tripwire/perseus.jhmg.pvt.twd TWReport perseus.jhmg.pvt 20170116032536 V:177 S:100 A:0 R:0 C:177
^^^ Auto run of Tripwire overnight showing 177 violations, all "changes"
  • Do you have automatic updates enabled, perhaps in /etc/cron.daily/yum.cron? It seems unlikely that so many binaries would have been modified, but there have been ELF infecting virii in the past. I would look at the most likely culprit first (updates somewhere via crontab) – movsx Jan 16 '17 at 19:45
  • Yep, a lot of integrity checkers will flip out after a system update. – multithr3at3d Jan 16 '17 at 19:52
  • That was my first thought too, but I don't have any auto-update service running. The system runs at runlevel 3 (no XWindows). I've added more information above: running services, syslog extract. – Jim Garrison Jan 16 '17 at 21:12
  • Check one of the smallest changed binaries to see what exactly is changed. That way you will see if it's a yum update or a virus. – Overmind Jan 17 '17 at 8:46
0

I can find no evidence of any updates being automatically applied, and both the yum and rpm databases concur that nothing was installed.

I think I found the culprit, and I feel pretty stupid now (hangs head in shame):

prelink

Carefully examining the list of files changed revealed that ONLY binaries were touched, not a single text file was modified. This is what would be expected if prelink did its thing and decided it needed to reorganize memory layout.

Since it runs every day and didn't cause this issue previously there must have been something in the packages I installed that made it take notice and decide to get to work. Maybe it was libnetfilter_conntrack or compat-readline5.

Anyway I think the problem is solved. If anyone disagrees, a comment or counter-answer would be appreciated. I'll wait a day or two before accepting just in case. I thought about deleting the question but it might be useful to someone else.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.