Validating security log configuration (rsyslog and syslog-ng) for PCI DSS 10.5

Question

I work for a Level 1 merchant with annual onsite QSA audits, so this is a huge deal.

PCI DSS 10.5 requires the entity under test to "Secure audit trails so they cannot be altered". In our Unix and Linux environment this requires that the syslog configuration be checked to see whether a) the permissions on all files used locally to store audit logs (defined as the auth, authpriv, and security facilities in syslog) are restricted to root, and b) these facilities are also exported to a log collector for safekeeping.

In the past, with classic syslog, we have managed this with a test of the syslog.conf file to determine which files to check permissions and which collector servers are in use. The issue is with the newer versions of syslog, namely rsyslog and syslog-ng.

The configuration files for these logging components are far more flexible and human-readable than their predecessors. However, we have been going quietly mad trying to validate configurations of the new tools for the following reasons:

In rsyslog (and syslog-ng) you can route messages on just about anything you can think of… including regular expressions. Not only can you route by those criteria, you could also completely discard certain messages based on them. For example, it would be easy enough to configure a system to look like it is logging all of the security stuff if one was simply looking through the logs, but all the while it is silently discarding any log message that contained a specific user ID. Easy to configure, difficult to detect.

With syslog-ng we have tried validating "known good" configurations with a checksum (this could be made more secure with a proper HMAC such as SHA-256, but that doesn't solve the underlying problem). The issue is that valid configurations tend to proliferate and it becomes impossible to keep up.

Both tools are capable of using include files, and rsyslog in particular uses this feature extensively "out of the box", which further complicates matters because now you have to checksum/hash not only the original file but every file it includes.

One stopgap is to flag every system running rsyslog or syslog-ng and require the SA to attest that the logs are being sent to the proper places, but with tens of thousands of systems, that approach does not scale, nor does it provide good evidence for auditors.

In short, we have found no technical solution to the problem of knowing for certain where every authentication log is being stored or transmitted, and yet PCI DSS 10.5 requires it. Therefore I am asking the security community what they are doing to facilitate this.

Answer 1

It seems to me like you're overanalyzing the issue - PCI DSS doesn't require you to ensure that the system is impossible to abuse, it requires you to perform due diligence along some basic abuse vectors.

Cites here are from DSS 3.1 "Testing Procedures":

10.5.1 Only individuals who have a job-related need can view audit trail files.

Ensure that auth* type logs are readable only by root. That can be pretty simple:

$ grep authpriv /etc/rsyslog.conf
# The authpriv file has restricted access.
authpriv.*                                              /var/log/secure
$ ls -l /var/log/secure
-rw------- 1 root root 1085793 Nov 10 13:43 /var/log/secure
$

And you're done. Is it possible that a malicious administrator could go out of their way to log authpriv somewhere else and have it be obscure to see where, and to also have that be world readable? It's possible, but that's not what the QSA is looking for, they're looking to ensure you didn't misconfigure your logs by mistake so everyone can read authpriv.

(edit) The situation you describe in the comment:

The issue with checksumming was that applications had valid needs to alter the configuration (in places that do not affect security logs) and that broke the checksum.

should actually be easier to address with rsyslog:

• system logs are specified in /etc/rsyslog.conf, standardized and auto-maintained across systems by cfengine.
• rsyslog.conf includes /etc/rsyslog.d/*.conf
• application log configurations go into /etc/rsyslog.d/application.conf

In this manner, you can easily prove that your system configuration for logging (say) authpriv is correct, and that the configuration is common across systems. The per-application customizations are isolated to the /etc/rsyslog.d/*.conf files, which you don't need to checksum or prove to your QSA - he may look at one or two, but in general, he will trust you as an administrator not to hide a redundant bad system-level configuration in the application-level config files.

Again - PCI DSS wants you to document you're doing basic things right. They're not trying to force you to prove, beyond a shadow of a doubt, that there's no way for you to do anything wrong.

10.5.2 Current audit trail files are protected from unauthorized modifications via access control mechanisms, physical segregation, and/or network segregation.

Show that your log files aren't writable to any inappropriate groups or 'other':

# find /var/log -type f -a \( -perm -g+w -or -perm -o+w \) -ls
131282 2220 -rw-rw-r--   1 root     utmp      2265216 Oct 30 09:51 /var/log/wtmp-20151101
133888  192 -rw-rw-r--   1 root     utmp       192000 Nov  9 16:12 /var/log/wtmp
# 

wtmp is group-writable to utmp to allow it to be updated by non-root terminals, so that's normal. No other findings. Along with basic checks to show that your logs go to /var/log, you're all set.

10.5.3 Current audit trail files are promptly backed up to a centralized log server or media that is difficult to alter.

10.5.4 Logs for external-facing technologies (for example, wireless, firewalls, DNS, mail) are written onto a secure, centralized, internal log server or media.

First, show the configuration indicating that logs are sent to your central SIEM. Then show a query proving that logs from your machine can be found on your SIEM. Repeat for whatever sample size your QSA requests.

You don't need to do a full reconciliation that every line found locally is found on the remote server. You just need to prove that you've configured it to log to a remote server, and that it apparently functions properly.

10.5.5 Examine system settings, monitored files, and results from monitoring activities to verify the use of file-integrity monitoring or change-detection software on logs.

Confirm that /etc/rsyslog* and /var/log/* are monitored by your FIM (e.g., Tripwire). That's it.

Are these steps enough to secure your environment? No! The PCI DSS is a floor, not a ceiling. It's a basic, due-diligence checklist to get you started and to try and ensure everyone is at least buckling their seat belts. So you can keep looking for ways to improve your security, but you'll likely pass your PCI audit on your way there.

Now, that being said, I'm detecting another issue in your question, which is that you lack centralized configuration control. That's going to make your audits harder - when you say "here's the config file, and here's the Puppet config showing it's used by 95% of our servers" you make it easier for the QSA to sign off on a sample of your environment. If you've got "valid configurations [that] tend to proliferate" you've got a problem that's not just impacting your PCI compliance, it's going to impact your overall security. You should look into centralized management (here's a comparison of four of such tools).

Answer 2

Very smart and creative people tend to think of all the eventualities that could break a system and account for them. That's a good instinct for creating a good, reliable system. But that's now what you're doing here. Don't confuse passing the PCI audit with creating an unbreakable security system, or even a really great security system. The two are very different. As gowenfawr mentions, all you really need to do is maintain due diligence, and not do really stupid things like have the log file alterable by anyone with access to the system.

You can't possibly create the perfect system like you're describing that's unalterable. At the very least a skilled attacker could simply replace your syslog with his own version that filters out certain records. With low level access to a system there's countless ways to cover your tracks. There's ways to detect that, but there's always going to be a way around it. Nothing is "unalterable" the way you're thinking of the word unalterable.