Is it advisable to fully record terminal interaction for incident response?

Question

We are going to be using a jump box to serve as an SSH bridge to our production systems. Assuming that operators are actually logging in to a shell on the jump box and assuming that it is possible to record the full terminal interactions, is it advisable to record full terminal interactions in this kind of scenario?

I am thinking that having those interactions recorded would serve as a great tool for postmortem reports and possibly for training purposes too. However, in what documentation I've read about computer incident response, I have never seen any mention of recording the terminal during the incident.

Answer 1

Yes, it's definitely advisable. In analysing an incident, you'll always want to have as much information as possible. Having too much information is better than not having enough as you don't want to be guessing links between actions.

For reviewing an incident, you'll want to know who did what when. In order to do this, you'll want to know the authentication so you know the who, you'll want the commands entered so you know the what and you'll need NTP sync'd so you know the when.

Depending on the system, some activity may not be captured in full when a user with privileges drops to a shell prompt. In these instances you may want to implement additional controls to prevent this or add software to detect it.

Answer 2

I recommend it also. For full terminal interaction I use the pam_tty_audit PAM module.

Just put this:

session     required      pam_tty_audit.so enable=*

at the end of your /etc/pam.d/system-auth-ac (RedHat/Fedora/CentOS) and you will start getting all keystrokes logged into /var/log/audit/audit.log.

The only downside to this (potentially a serious one) is that it logs all tty interaction including passwords. This issue is being addressed at my request.

Answer 3

Yes, there are many reason why you would want to have a complete audit trail of everything done on a given server (or everything you do on any server), and auditing is just a small fraction of it.

Often information that later proves critical is only ever dumped to the console (not to any log file), and having a history to go look through can be very helpful.

Also it's useful in a CYA context ("Who deleted the passwd file yesterday?") and in troubleshooting recurring issues.

You can record your own terminal sessions using script, which you can later examine. I wrote a bash script I called scriptssh which I use on my local box when I want to keep a log of an ssh session. I just type scriptssh instead of ssh to make the connection:

#!/bin/bash
# TODO: can choke on commands containing quotes.

BASE_DIR=$HOME/ssh-archive/
DT=`date +%Y-%m-%d`
TM=`date +%H.%M.%S`

# new directory for each day (makes filesystem much faster)
DIR=$BASE_DIR/$DT
[[ -d $DIR ]] || mkdir -p $DIR

# Sanitized for your protection
ARGLIST=$(sed 's/[^A-Za-z0-9.-]/_/g' <<< "$*")
# build filename based on date and passed-in arguments
FNAME=$DIR/$TM--$ARGLIST

# make sure filename is unique
if [[ -f $FNAME ]]; then
    $I=0
    while [[ -f $FNAME.$I ]] ; do (( $I += 1 )); done
    FNAME=$FNAME.$I
fi

exec script -a -c "/usr/bin/ssh $*" $FNAME