Why is it possible for the root user to delete the logs?

Question

I always thought the greatest benefit of the logs is to confirm to you that your machine has been hacked. However I see hackers bragging about "rooting" servers all over the Internet. Whats stopping a hacker with root access from deleting the logs in order to cover their tracks?

Answer 1

Why is it possible for the root user to delete the logs?

Because you need to be able to manage the server.

Whats stopping a hacker with root access from deleting the logs in order to cover their tracks?

Regarding local logging, only their own stupidity. Regarding external logging, their skill set and yours.

There are many, many different ways to "log." In fact, you could have the rooted machine log to a separate, hidden device that can't easily be overwritten, such as a raspberry pi / arduino logger, of which I have a few.

In Linux, root is supposed to be able to do pretty much everything. In Windows, the same concept applies; if you have root access to a machine, you will be able to do almost anything you want to that machine, which includes deleting logs.

You can also log all connections to a certain machine, especially if it isn't supposed to be "accessed" in the first place.

How would you find out what's going on, and how would you circumvent it?

Here's a way to find all changes in the last 120 minutes:

find / -mmin -120 -printf '%p\t%a\n'

Nice, huh? Gotcha, hacker! Mwahahaha... but you can change these attributes if you know what you're doing, like so:

touch -d "24 hours ago" <file>

Even worse, automating it:

find / -print | while read file; do
    touch -d "$(date -r "$file") - 24 hours" "$file"
done

But wait, Mark Hulkalo! If all the files were last edited 24 hours ago, then that clearly shows we've got a haxor! True, but you can also apply $RANDOM where 24 exists. Here's an example of a number from 1-24: $((RANDOM % 10) + 1). Now it doesn't look so easy to discover, does it?

But what if all files look like that, especially files which aren't supposed to be changed? Limit the scope to a certain directory, such as /root. There are lots of ways to mask your presence locally.

Unless the attacker forgot to cleanse ~/.bash_history, you'll be flying blind. Really, if the only "logging" you have is ~/.bash_history, then you're in trouble if your attacker is at least moderately intelligent. That can be erased on each user account with astounding ease, so as long as you have the appropriate permissions.

It's much easier to log entries somewhere it won't be easily detected or modified, such as an external device. While it's possible to use forensics to recover your attacker's footprints, it's very, very easy to circumvent if you know what you're doing. This is why external logging is a much better solution.

And that's the story of why external logging is generally the answer... keep in mind there are also ways to circumvent external logging. Nothing is ever 100% safe; you can only make it harder for your attacker, not impossible.

Answer 2

Why is it possible for the root user to delete the logs?

Because somebody must be able to make changes to the system, i.e. rotate log files, remove log files etc. In UNIX root is traditionally this fully privileged user. Actually there is not much of a difference between modifying a log and replacing a binary to omit logging - and root can do both by default.

Whats stopping a hacker with root access from deleting the logs in order to cover their tracks?

The standard configuration usually does not stop the attacker at all. One obvious way to improve this is logging to a remote logserver which is protected enough so that the attacker has no access (in the simplest case: no remote login possible).

Apart from that FreeBSD, OpenBSD and probably others have the idea of immutable or append-only files. With chflags you could mark log files this way and then only appends are possible and the file can not be deleted. Combining this feature with securelevel and you can achieve log files which can only be changed (removed, rotated) in single-user mode.

And you could use alternative or extended security mechanisms like SELinux to make sure that the "normal" root user cannot change files.

Answer 3

If the server has had the root user compromised, that user can certainly delete any files (including logs) on the systems. But, in most enterprises, logs aren't stored locally.

If logs are stored locally and deleted, in order to determine the extent of the breach, forensic analysis can be conducted to recover the files.

However, most enterprise web servers and corporate environments use some type of log management -- this can range from standard tools such as Syslog and SNMP, and proprietary log management software like Splunk.

When those systems are used, the attacker would have to compromise the log management system and any backups thereof in order to delete the logs and cover their tracks.

So sure, one can as root delete files; but this isn't a certain way of covering one's tracks.

Answer 4

Because WORM (write once, read many) devices are still uncommon. In systems where security and forensics are very important these devices are used to record logs in real time, but cannot be deleted from, even by root. To be able to implement this securely it has to be a hardware limitation. Less securely it could expose an API to a software service which only has operations to write new files and read old ones. If the server also has a hardware device or hypervisor which logs basically every change to the system it would be very difficult to hide your tracks.

Answer 5

For the truly paranoid, something along the lines of:

tail -f /var/log/auth.log > /dev/lp0

... would probably stop all but those who would stoop to arson.

Whip out the Oki ribbons!

Answer 6

Because SOMEBODY has to be able to manage log files, and if not root, then who? Would you really want a system where if, say, a log file grows to take up 90% of your disk space, there is absolutely no way to delete it?

Sure, this is a security hole. In the same sense that a system that allows any user to do anything creates a security hole. Most door locks are designed so that the lock can only be removed from the door by someone who is inside the house. But this means that anyone who can get inside your house, such as a guest or a plumber, could replace your door lock with one to which only he has the key. Is this a security hole? Sure. But what's the alternative? Make it so that no one can replace the lock? What if the lock breaks?

In real life, it's not a simple either/or: "insecure systems" where anybody can get in and do anything, versus "secure systems" where we are 100% certain that only authorized users can do authorized things. There's a range of "secureness". Often, the more secure you make a system, the more inconvenient it becomes for authorized users to do their jobs. At some point security is so tight that the system is virtually unusable.

Answer 7

I've kind of said this in comments but it seems it deserves a full answer.

If you're using SELinux, you can configure it in such a way so that root cannot delete log files. SELinux uses Mandatory Access Control (control based on roles) in order to determine which roles can read/write/execute each file, on top of Linux's Discretionary Access Control which states what each user/group/everyone can do to a file - the typical rwxrwxrwx permissions.

There seems to be confusion about how SELinux might be used to accomplish this, but it's pretty straight forward. One possible point of confusion is the idea of a Linux user and an SELinux role. Nothing changes as far as Linux users go. SELinux only adds on top of what Linux already has. So you'd setup your users as normal. root would be the same as always as well.

Then you need to introduce SELinux roles. Linux users can be assigned many SELinux roles, and a user's roles determine what it can do on the system. Now, the default SELinux policy - 'targeted' I think is what it's called - has an unconfined user, which is analogous to a superuser. By default, root is assigned as unconfined, so root can do anything just like it could without SELinux. I imagine most people don't modify the default policy or the default roles in SELinux, so it gives the impression that root can do whatever it wants.

However, this isn't necessary. If you take away root's SELinux role, then it can't do anything(you wouldn't even be able to log in). Before doing this, however, you would want to setup a user with the role of administrating SELinux. Both 'targeted' and 'mls' policies come with such administration roles built in, but I forget what it's called exactly. You'd typically give this to one of your system administrators so that they can modify SELinux policy, roles, etc if needed.

Your question hits on the entire point to SELinux and its mandatory access system. People realized that getting root gives hackers far too much. They can screw up your ssh, your httpd, and anything else they want to. But with SELinux you can restrict or remove root access, so even if someone logs in to root they can't just destroy your system. Note: this applies to more than just log files. This applies to everything on the system. Getting root becomes meaningless very quickly if you configure SELinux properly.

And to be clear - some user at some point has to be able to delete the log files. The point of this isn't to make deleting log files impossible, the point is to make it impossible for root to delete any/all log files. If someone/something on the system can make a log file, then they can also delete that log file, unless you're using some sort of write-once memory like others have mentioned.

So then the question shifts a little bit: a hacker compromises your web service, and can subsequently delete the logs associated with it, right? And that's really what we care about, because that's the log that will trace their actions. If they can still modify that, then what's the point of this whole circus act? Well, the answer is they might be able to delete/alter the log files, depending on what context they're running in once they've exploited the system, and if that context has permission to modify the relevant log files. But that's a rabbit hole in itself and this answer is long enough as-is. Still, SELinux will be recording system calls anyways, and the hacker who compromised your web service won't be able to modify SELinux logs (they're two separate contexts/roles on the system), so you'll at least have that.

Answer 8

By default, nothing stops an attacker from deleting the logs to cover their tracks, however, if an attacker deletes the logs it will be obvious that something has happened. Something more intelligent from an attacker point of view is modifying the logs to just delete his/her tracks.

In the other hand, you can implement some measures to protect the logs, for example, send them in real time to an external server.