Comparing two Linux system clones - what about non-regular files?

Question

Could you please help me to understand the best approach to compare two filesystems / hard disks?

As a practical learning exercise I created a clone of whole hard drive a month ago and again yesterday (it’s Ubuntu Server and I cloned it on Debian just using DD on disconnected hard drive). The point is to compare known state and unknown that was potentially compromised.

After DDing (sudo dd if=/dev/sdX of=/tmp/my_image1) I attached both clones:

sudo losetup --partscan --find --show /tmp/my_image1

I changed FSID so I can mount it and I mounted it:

sudo mount /dev/loop0p3 /mnt/0a -o ro

sudo mount /dev/loop1p3 /mnt/0a -o ro

After that I simply compared both filesystems to find possible change / malware:

sudo diff --no-dereference --brief --recursive /mnt/0a /mnt/0b

It was just for learning, I didn’t assume to find anything else then new logs, Bash history...

But strange thing that I discovered are non-regular files:

sudo find /mnt/0a -not -type f,d,l -exec ls -l '{}' \;

crw-rw-rw- 1 root root 5, 1 Apr 23 07:32 /mnt/0a/dev/console
crw-rw-rw- 1 root root 1, 7 Apr 23 07:32 /mnt/0a/dev/full
crw-rw-rw- 1 root root 1, 3 Apr 23 07:32 /mnt/0a/dev/null
crw-rw-rw- 1 root root 5, 2 Apr 23 07:32 /mnt/0a/dev/ptmx
crw-rw-rw- 1 root root 1, 8 Apr 23 07:32 /mnt/0a/dev/random
crw-rw-rw- 1 root root 5, 0 Apr 23 07:32 /mnt/0a/dev/tty
crw-rw-rw- 1 root root 1, 9 Apr 23 07:32 /mnt/0a/dev/urandom
crw-rw-rw- 1 root root 1, 5 Apr 23 07:32 /mnt/0a/dev/zero
crw------- 1 root root 10, 236 Apr 23 07:34 /mnt/0a/dev/mapper/control

Could anyone please help me understand:

1) Hurpose of character devices on disconnected hard drive - system is not running, I thought that these files are created by system when it’s running (like files in /prod and /dev) and the purpose is to interface with the system, not to store data

2) How am I suppose to compare it? I can compare standard file (bit by bit), I can compare directory names and I can compare symbolic link (by comparing targets) but I have no idea how to compare this.

Answer 1

1) Hurpose of character devices on disconnected hard drive - system is not running, I thought that these files are created by system when it’s running (like files in /prod and /dev) and the purpose is to interface with the system, not to store data

While /proc is completely generated by the kernel, /dev does have some real contents. While a devtmpfs does usually get mounted over it, early boot tasks that run before that happens need some of the devices. Even though character devices are to interface with the system, they still exist on disk; they just don't store their contents there.

2) How am I suppose to compare it? I can compare standard file (bit by bit), I can compare directory names and I can compare symbolic link (by comparing targets) but I have no idea how to compare this.

Compare the major and minor device numbers (e.g., 1, 3 for /dev/null). If two character device files have the same device numbers, then they point to the same device and can thus be considered identical. If not, then they point to different devices and should be considered different.