I always thought the greatest benefit of the logs is to confirm to you that your machine has been hacked. However I see hackers bragging about "rooting" servers all over the Internet. Whats stopping a hacker with root access from deleting the logs in order to cover their tracks?
Why is it possible for the root user to delete the logs?
Because you need to be able to manage the server.
Whats stopping a hacker with root access from deleting the logs in order to cover their tracks?
Regarding local logging, only their own stupidity. Regarding external logging, their skill set and yours.
There are many, many different ways to "log." In fact, you could have the rooted machine log to a separate, hidden device that can't easily be overwritten, such as a raspberry pi / arduino logger, of which I have a few.
In Linux, root is supposed to be able to do pretty much everything. In Windows, the same concept applies; if you have root access to a machine, you will be able to do almost anything you want to that machine, which includes deleting logs.
You can also log all connections to a certain machine, especially if it isn't supposed to be "accessed" in the first place.
How would you find out what's going on, and how would you circumvent it?
Here's a way to find all changes in the last 120 minutes:
find / -mmin -120 -printf '%p\t%a\n'
Nice, huh? Gotcha, hacker! Mwahahaha... but you can change these attributes if you know what you're doing, like so:
touch -d "24 hours ago" <file>
Even worse, automating it:
find / -print | while read file; do touch -d "$(date -r "$file") - 24 hours" "$file" done
But wait, Mark Hulkalo! If all the files were last edited 24 hours ago, then that clearly shows we've got a haxor! True, but you can also apply
$RANDOM where 24 exists. Here's an example of a number from 1-24:
$((RANDOM % 10) + 1). Now it doesn't look so easy to discover, does it?
But what if all files look like that, especially files which aren't supposed to be changed? Limit the scope to a certain directory, such as
/root. There are lots of ways to mask your presence locally.
Unless the attacker forgot to cleanse
~/.bash_history, you'll be flying blind. Really, if the only "logging" you have is
~/.bash_history, then you're in trouble if your attacker is at least moderately intelligent. That can be erased on each user account with astounding ease, so as long as you have the appropriate permissions.
It's much easier to log entries somewhere it won't be easily detected or modified, such as an external device. While it's possible to use forensics to recover your attacker's footprints, it's very, very easy to circumvent if you know what you're doing. This is why external logging is a much better solution.
And that's the story of why external logging is generally the answer... keep in mind there are also ways to circumvent external logging. Nothing is ever 100% safe; you can only make it harder for your attacker, not impossible.
but how do you erase from the base_history the command you used to erase bash_history?– njzk2Nov 11, 2015 at 4:17
2@njzk2 history -c doesn't leave itself. or just delete .bash_history. or copy it then do your thing then replace it with the original. or just don't erase it; your other tracks are still covered.– Jason CNov 11, 2015 at 4:23
4Or you can use
export HISTFILE=/dev/nullbefore you start typing your commands. Nov 11, 2015 at 7:37
2(in bash at least) putting a space before a command hides it from the bash history. Bash history is not intended as a security feature. Nov 11, 2015 at 20:37
1The default on Debian is HISTCONTROL=ignoreboth. There are other ways to hide things from bash history - e.g ssh target command-to-run. No matter what you do, it is still not a security feature, and should not be depended on. Nov 12, 2015 at 0:55
Why is it possible for the root user to delete the logs?
Because somebody must be able to make changes to the system, i.e. rotate log files, remove log files etc. In UNIX root is traditionally this fully privileged user. Actually there is not much of a difference between modifying a log and replacing a binary to omit logging - and root can do both by default.
Whats stopping a hacker with root access from deleting the logs in order to cover their tracks?
The standard configuration usually does not stop the attacker at all. One obvious way to improve this is logging to a remote logserver which is protected enough so that the attacker has no access (in the simplest case: no remote login possible).
Apart from that FreeBSD, OpenBSD and probably others have the idea of immutable or append-only files. With chflags you could mark log files this way and then only appends are possible and the file can not be deleted. Combining this feature with securelevel and you can achieve log files which can only be changed (removed, rotated) in single-user mode.
And you could use alternative or extended security mechanisms like SELinux to make sure that the "normal" root user cannot change files.
1Been a while since I've used FreeBSD / OpenBSD, but what happens when you do this?
chflags nouchg file. And I'm almost fairly confident that securelevel can be circumvented. Nov 10, 2015 at 18:55
1Linux has immutable files too, and you can permanently revoke the right to change the immutable flag on any file until rebooted... this makes certain mechanisms at least hard to tamper with. You need to revoke further rights to make that really bulletproof though - if root can change memory or block device contents directly.... so logging to a remote service, or to true append-only media (eg a printer, cd-r, tape that cannot be made to rewind from the host) is the only really safe thing there... Nov 10, 2015 at 19:26
@MarkHulkalo: I have no idea of uchg but schg can only cleared by the superuser and only when running in security level 0 or -1 (see documentation. And I doubt that you will be able to decrease the securelevel once it is set without a reboot. Nov 10, 2015 at 19:29
1@rackandboneman: Linux has immutable flag but linux has not the simple securelevel concept. Instead they use capabilities and by default root has all of them - which means that root can remove the immutable flag again. Nov 10, 2015 at 19:31
@Steffen Ullrich root can drop the capability to set or unset the flag - permanently Nov 10, 2015 at 20:18
If the server has had the root user compromised, that user can certainly delete any files (including logs) on the systems. But, in most enterprises, logs aren't stored locally.
If logs are stored locally and deleted, in order to determine the extent of the breach, forensic analysis can be conducted to recover the files.
However, most enterprise web servers and corporate environments use some type of log management -- this can range from standard tools such as Syslog and SNMP, and proprietary log management software like Splunk.
When those systems are used, the attacker would have to compromise the log management system and any backups thereof in order to delete the logs and cover their tracks.
So sure, one can as root delete files; but this isn't a certain way of covering one's tracks.
Because WORM (write once, read many) devices are still uncommon. In systems where security and forensics are very important these devices are used to record logs in real time, but cannot be deleted from, even by root. To be able to implement this securely it has to be a hardware limitation. Less securely it could expose an API to a software service which only has operations to write new files and read old ones. If the server also has a hardware device or hypervisor which logs basically every change to the system it would be very difficult to hide your tracks.
Do these WORM devices probably just use DVD+-R's, or something more exotic/expensive?– Xen2050Nov 11, 2015 at 9:24
Some magneto-optic drives were WORM, but I haven't seen one of them in years.– Simon BNov 11, 2015 at 11:07
@Xen2050 a DVD-R burning session takes minutes, which may well be enough for the attacker to remove the traces, so the next DVD will not contain them. Nov 11, 2015 at 17:21
@DmitryGrigoryev I'd assume they would use a custom writing format, so no waiting for a whole "session to close" or anything like that. I remember old CD-R software that would treat the whole disc like a big floppy and, I think, start writing files in a few seconds, it was only "finishing" the whole disc that took a few minutes. Packet writing it's called. (And replace DVD-R with BD-R XL, or whatever the current big-disc letters are in the future)– Xen2050Nov 11, 2015 at 18:05
@Xen2050 I believe packet writing only seems to happen within seconds because of caching, and ejecting the disk before the session is closed would result in data loss. Nov 11, 2015 at 18:17
For the truly paranoid, something along the lines of:
tail -f /var/log/auth.log > /dev/lp0
... would probably stop all but those who would stoop to arson.
Whip out the Oki ribbons!
1E.g. en.wikipedia.org/wiki/The_Cuckoo%27s_Egg Nov 11, 2015 at 20:23
Because SOMEBODY has to be able to manage log files, and if not root, then who? Would you really want a system where if, say, a log file grows to take up 90% of your disk space, there is absolutely no way to delete it?
Sure, this is a security hole. In the same sense that a system that allows any user to do anything creates a security hole. Most door locks are designed so that the lock can only be removed from the door by someone who is inside the house. But this means that anyone who can get inside your house, such as a guest or a plumber, could replace your door lock with one to which only he has the key. Is this a security hole? Sure. But what's the alternative? Make it so that no one can replace the lock? What if the lock breaks?
In real life, it's not a simple either/or: "insecure systems" where anybody can get in and do anything, versus "secure systems" where we are 100% certain that only authorized users can do authorized things. There's a range of "secureness". Often, the more secure you make a system, the more inconvenient it becomes for authorized users to do their jobs. At some point security is so tight that the system is virtually unusable.
Yes but why the superuser? why not let this aspect managed by an other user in order to make it difficult for the hacker to erase his tracks?– UlkomaNov 11, 2015 at 19:47
Okay, so suppose instead of root manipulate the logs, you create some other user, "log_manager", who can manipulate the logs. How does this help? Then if a hacker can log in as log_manager, he can cover his tracks. How is this better than requiring him to log in as root to do this? If root has authority to change log_manager's password, then if the hacker can get in as root, he can get in as log_manager. I suppose you could create many users, each able to do only a small set of "admin" things, and with no one user able to manage all their IDs and passwords, but like only Al can update Bob, ...– JayNov 11, 2015 at 20:13
... only Bob can update Cathy, only Cathy can update David, etc. Still, if the hacker can get in at any point in the chain, he could chase it down. It would be harder, and if security needs were big enough -- if we're talking plans for nuclear weapons or my credit card number -- maybe it's worth it. But it's still not 100% secure, it's just more work to hack.– JayNov 11, 2015 at 20:15
why cant we have something like the SAM files in MSWindoes, even the administrator can't modify them, only the system user can. This means if you want to mess with them you need to boot the server from an external device. How difficult it is to implement such thing??– UlkomaNov 11, 2015 at 20:19
I've kind of said this in comments but it seems it deserves a full answer.
If you're using SELinux, you can configure it in such a way so that root cannot delete log files. SELinux uses Mandatory Access Control (control based on roles) in order to determine which roles can read/write/execute each file, on top of Linux's Discretionary Access Control which states what each user/group/everyone can do to a file - the typical rwxrwxrwx permissions.
There seems to be confusion about how SELinux might be used to accomplish this, but it's pretty straight forward. One possible point of confusion is the idea of a Linux user and an SELinux role. Nothing changes as far as Linux users go. SELinux only adds on top of what Linux already has. So you'd setup your users as normal.
root would be the same as always as well.
Then you need to introduce SELinux roles. Linux users can be assigned many SELinux roles, and a user's roles determine what it can do on the system. Now, the default SELinux policy - 'targeted' I think is what it's called - has an unconfined user, which is analogous to a superuser. By default, root is assigned as unconfined, so root can do anything just like it could without SELinux. I imagine most people don't modify the default policy or the default roles in SELinux, so it gives the impression that root can do whatever it wants.
However, this isn't necessary. If you take away root's SELinux role, then it can't do anything(you wouldn't even be able to log in). Before doing this, however, you would want to setup a user with the role of administrating SELinux. Both 'targeted' and 'mls' policies come with such administration roles built in, but I forget what it's called exactly. You'd typically give this to one of your system administrators so that they can modify SELinux policy, roles, etc if needed.
Your question hits on the entire point to SELinux and its mandatory access system. People realized that getting root gives hackers far too much. They can screw up your ssh, your httpd, and anything else they want to. But with SELinux you can restrict or remove root access, so even if someone logs in to root they can't just destroy your system. Note: this applies to more than just log files. This applies to everything on the system. Getting root becomes meaningless very quickly if you configure SELinux properly.
And to be clear - some user at some point has to be able to delete the log files. The point of this isn't to make deleting log files impossible, the point is to make it impossible for
root to delete any/all log files. If someone/something on the system can make a log file, then they can also delete that log file, unless you're using some sort of write-once memory like others have mentioned.
So then the question shifts a little bit: a hacker compromises your web service, and can subsequently delete the logs associated with it, right? And that's really what we care about, because that's the log that will trace their actions. If they can still modify that, then what's the point of this whole circus act? Well, the answer is they might be able to delete/alter the log files, depending on what context they're running in once they've exploited the system, and if that context has permission to modify the relevant log files. But that's a rabbit hole in itself and this answer is long enough as-is. Still, SELinux will be recording system calls anyways, and the hacker who compromised your web service won't be able to modify SELinux logs (they're two separate contexts/roles on the system), so you'll at least have that.
By default, nothing stops an attacker from deleting the logs to cover their tracks, however, if an attacker deletes the logs it will be obvious that something has happened. Something more intelligent from an attacker point of view is modifying the logs to just delete his/her tracks.
In the other hand, you can implement some measures to protect the logs, for example, send them in real time to an external server.
setenforce 0to disable SELinux and then deletes the logs. Now what? It is very hard to implement - if you make another user to delete logs, root can
suto that user. If you have a part of the system that root cannot get to, how can it be managed or configured? Then you just move the hacker from trying to "gain root access" to "trying to gain 'secure area' access" instead.