After checking a vulnerable linux machine for rootkit and deleting them, we need to get some informations about the process , port ,incoming or outgoing connections... , using some useful commands as ps , netstat , top, lsof ...

Can we trust the information displayed by the linux utility commands?

  • 1
    only if you confirmed that the commands were the standard commands – schroeder May 11 '16 at 18:50
  • 9
    And maybe not even then. These commands depend on libraries which might be exchanged. They also depend on the kernel to report the correct information, which is not guaranteed if the system is compromised. – Steffen Ullrich May 11 '16 at 18:59
  • 18
    "Can we trust […] a vulnerable machine?" – No. That's really pretty much always the answer to any question of that form. – Jörg W Mittag May 11 '16 at 20:17
  • 2
    Since it's specifically a rootkit, the answer is no. Rootkits are generally designed to evade detection by fooling such utilities. – Monstieur May 12 '16 at 5:09
  • 1
    @drewbenn Not a duplicate, but a complementary question/answer. – Shadur May 12 '16 at 8:42

As the system is compromised, nothing is to be trusted via way of tools. Unless you have the tools validated (e.g. Tripwire FIM), your best bet is to take a similar system, copy over what is necessary, which should run if the systems are similar in architecture, etc. This is not the optimal method though. Because the machine is compromised, depending on your next steps (legal, authorities, etc), you would create a forensic image, then deal with what occurred as you have your copy. Once you have your copy, you need to determine the risk associated with putting the system back online, etc.

If you have determined how an attacker got onto the system, you would need to clean up that 'hole' (vulnerability, misconfiguration) in order to be sure they don't return. Sometimes this can be more time consuming versus installing a clean system. But let's say you need 'that' system. You could reinstall ps with something like: apt-get install --reinstall procps same applies for lsof. You would want to make sure your repos were not changed, and your DNS is not pointing to a non-trusted repo.

For the most part to answer your question: Can we trust the information displayed by the linux utility commands the answer is you absolutely should not. Little on that system should be trusted until a thorough analysis is performed.

  • It seems unlikely that there's anything specific in the system that would require it to be reused. Even if you must use the hardware, you could copy any files needed for recovery, analyze them offline for tampering/infection, wipe the hard drive, and then reinstall the OS (since it's a common OS and not some embedded thing you can't get anywhere) and packages from scratch before putting your application and data back in place. No? – jpmc26 May 12 '16 at 3:18
  • Reinstalling procps won't help if your kernel is still compromised. Unless they actually broke ps, it probably still works correctly, except for not showing the attacker's hidden processes. – Peter Cordes May 12 '16 at 5:43
  • 6
    "You could reinstall ps with something like: apt-get install --reinstall procps" - note that also apt-get may be compromised and not working as expected. – reducing activity May 12 '16 at 5:54

If your system has been compromised, you shouldn't trust anything.

I think usually the standard utilities will mostly work correctly, but leave out stuff related to the attacker's processes. Rootkits are designed this way so you're less likely to notice that the machine is compromised. So I think you can generally trust them for looking at your own processes, but not for making sure a rootkit is gone.

If the attacker can load kernel modules, or otherwise modify the kernel, even the system calls and /proc API can lie. So even a clean copy of the user-space utilities like ps, or grep foo /proc/*/cmdline, won't tell you if there's a malicious process running. Any rootkit worth its salt will hide its own processes.

Every file on the entire system is like radioactive waste, that can potentially contaminate other stuff if you aren't careful. e.g. an attacker might have added something to /home/*/.bashrc to re-infect your system in case you reinstall the OS but don't check /home.

Similarly, there can be nasty stuff in your web server config, or in your CGI scripts, etc. Compare against backups, and don't assume anything is safe if the attacker could have touched it.

Definitely do any and all checking of untrusted data on a known-clean machine. As long as you don't run anything from the compromised system, you should be ok. (i.e. assuming cmp and diff don't have any vulnerabilities. But note that strings isn't safe on untrusted files, depending on the version of libbfd. Use strings -a.


Probably, but not necessarily. The attacker could always replace the programs with modified versions of their own if they had root access.

  • 6
    Not just "could", but "often will". Lots of rootkits do this out-of-the-box, so I wouldn't say "probably" if your attacker is at all competent. – Charles Duffy May 11 '16 at 21:11
  • 2
    I think "could" suffices here. – user1717828 May 11 '16 at 23:25
  • 2
    If the attacker can load kernel modules, or otherwise modify the kernel, even the system calls and /proc API can lie. So the lack of trustworthiness extends beyond user-space tools and into the kernel itself. – Peter Cordes May 12 '16 at 5:30

Amazingly, against all the laws of the universe, a car-based analogy is useless here.

An Invasion of the Body Snatchers analogy works, though.

Any of your system's commands (or the libraries that they depend upon) can be (and probably has been) replaced with a copy that looks and acts almost exactly like the original but also has the secret purpose of hiding the existing compromise and/or assisting any future compromise attempt.

In short, the answer is "No, you can not trust any of the programs on your compromised system".

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.