Information Security Stack Exchange is a question and answer site for information security professionals. It only takes a minute to sign up.
Sign up to join this community
Let's say you deploy an incident response tool on a server or endpoint such that you can connect to the system remotely if an incident occurs.
In one of the worst cases you could have a rootkit, wouldn't that in and of itself make one not trust what your incident response tool is telling you. Wouldn't you have to instead bring a CD with trusted (statically linked) tools to the host in question to be sure the commands you run are telling you the truth?
As an Incident Responder, you always want to be very careful on what assumptions you make when you're on a system. In that sense, your questions are quite pertinent.
However, you also exercise your judgment (in some cases, wrongly perhaps) on how much care is required in a particular instance. There are limits on what you can do with public-cloud-hosted systems. You try and work with those limits (such as not being able to boot from a CD); and find workarounds where possible. e.g., if you don't need live forensics, you could create your own "trusted boot image with trusted tools", boot from that image, attach the disks in question and investigate. This is pretty much the same principle as carrying you own boot disks (CDs or USB drives). I do this by default only when I'm working on highly sensitive servers - where the adversaries are likely to be highly skilled.
In all other cases I take the more efficient / pragmatic route - as @john-deters suggests.
Live Forensics are a bit more slippery. No easy how-tos there. You could mount a disk with your own tools (actually, in many cases you do) but that carries its own risks.
Finally, if you're going to install remote IR tools (like GRR for instance), you can only take precautions while installing the tools. As an added precaution, you should create a file-integrity alert (you need remote logging for this) to make sure the GRR package is not altered any way. If that isn't triggered, you are probably (yes probably, not certainly!) on solid ground when logging in for IR remotely.
In the real world, that's not practical. Amazon isn't going to let you in the door of their data center to insert a CD into one of their cloud computers. Has that stopped Amazon Web Services from existing? Has that stopped people from building many successful businesses in Amazon's cloud? The existence (and profitability) of AWS suggests that they're pretty trustworthy.
Instead, we try to install the software security agents on freshly built machines, and exercise a bit of trust that they weren't maliciously modified in the meantime. It's a practical approach that covers most of the attack surface.
Consider the inverse: How would you know if the CD you would use to install the agent is a genuine image, and not a clever forgery that contains a root kit? How do you know the server doesn't have a malicious chip created by an untrustworthy foundry, and installed by an unscrupulous motherboard vendor? How do you know some shadowy attacker hasn't installed a COTTONMOUTH-III board in the machine's USB tower while it sat waiting to clear Customs? At some point, you have to let go and exercise some trust. The trick is to never let the inability to achieve absolute perfection keep you from getting to a state that's "good enough". Yes, there's always a gap, a hole in the process, something that could be exploited by the right person at the right time. Minimize the gaps, certainly, but be aware they're going to exist regardless of what you do.
A more rational approach is to perform a risk assessment and risk analysis. You have a finite security budget -- everyone does. Do you want to spend it all on carrying CDs around to your servers on the tiny chance it's already infested with a rootkit that you're completely unaware of? Or would it make more sense to spend it on a more immediate threat or urgent problem, like implementing a network intrusion detection and response system, or training your users to recognize phishing attacks, or auditing your TLS configuration?
Hit the realistic problems first.
