Does anyone have a general step by step list on how to try discover rootkits on a Linux or Solaris server?
Step one: Disconnect the machine in question from all communications channels: Ethernet, WiFi, Bluetooth, serial, Firewire, USB, audio cable, Infrared, parallel. Only the power cable, keyboard cable, and monitor cable should remain connected. Check for RF emissions with a calibrated RF spectrum analyzer. Only proceed when you have verified communication disconnection.
Sep two: If the system has a CD-ROM or DVD-ROM drive and is capable of booting from the CD-ROM, on a know clean system prepare a cold boot RAM attack CD-ROM. Otherwise proceed to Step three. Freely available tools for cold boot attack: msramdump and Princeton.
Step three: If you prepared a cold boot attack disk in Step two do not power off the machine. Disconnect the hard drive(s). As each drive is disconnect connected label it with a brightly colored label. On the label write 'Infected', the time and date, the name of the system it was remove from. Place each drive into an appropriate storage container.
Step four: If you did not prepared a cold boot attack disk in Step two precede to Step six. Otherwise insert the cold boot attack disk and reboot the machine. Capture the contents of the RAM and save it a CD-R or DVD-R. Make sure to finalize the disk for every recording. After saving contents of RAM power down the machine.
Step five: On a separate know good machine analyze the RAM dump. Check for executable file format signatures. Check for virus signatures. Compare to a dump of a known good machine running the same operating system.
Step six: On your analysis system disable any automount, autoplay, plug and play, or any service that takes acts on detection of a new drive. Verify that the system takes no action on new drive attachment by attaching a known good drive of the same physical connector type as the drives from the infected system. If the infected system had drives of more than one physical connect type, then test the analysis system with a known good drive for each physical connector type. If there are multiple drives, only attach one drive at a time to the analysis machine. Create an image of each drive and then store the drives in appropriate containers.
Step seven: Begin analysis on the image of the primary boot drive. Using static analysis, check the boot sequence. Perform differential binary analysis on the primary operating system initialization components, including file meta-data structure and creation order.
Step eight: Begin analysis of security critical libraries.
Step nine: Begin analysis of system administration tools, logs, and configuration files.
Step ten: Begin analysis of services and daemons.
Step eleven: Begin analysis of applications.
Step twelve: Begin analysis of user files and remaining files.
Places to look for suspicious files?
Every file should be considered suspicious until you have verified that it is not.
Search commands to pick up weird file behavior?
No commands should be run on a potentially compromised machine. As soon as you suspect a system may be compromised it should be disconnected for all communication channels. If the system is not a candidate for a cold boot attack on the RAM, then it should be powered off immediately and the power cord removed.
Weird looking traffic picked up from sniffer?
No traffic should be sent or received on a potentially compromised machine.
Checking file integrity on important commands in order to know if you can trust it or not?
File integrity may be check on an analysis machine using read only drive images if you have a known good file integrity database. Every file in the database should be checked.