Heuristically. nmap observes the behaviour of the system during probes, develops a "fingerprint", which analagous to a real fingerprint test, looks mainly for the minutiae:
- out of specification behaviour and extra-specification or undefined behaviour
- retransmission times
- response to fragmentation and various ICMP probes
- patterns in TCP sequence numbers, IP ID numbers and TTLs
- behaviour with various TCP options and flags
That's just a quick summary, you can read a more thorough description here
The fingerprint is then compared with a database of known fingerprints. In the event that you scanned a system that has a unique fingerprint you will get a good match, otherwise you will get a guess based on the best fit (which is sometime wrong). It might be wrong anyway, some people like to play games (old but interesting). If there are no good matches,
nmap will encourage you to find out what the system is and submit it.
You can find the fingerprints in
/usr/share/nmap/nmap-os-db (probably), but you'll have to read the documentation to decipher the details.
In the current nmap database (6.47) there are 4485 fingerprints for 5009 CPE (platform) names, over 800 of which match various versions of MS-Windows, and over 1000 of which match various Linux systems (including appliances).
Here's an actual signature for a Linux system, an Asus Nexus 7 tablet:
# Linux 3.1.10-g52027f9 #1 SMP PREEMPT Thu Jun 28 16:19:26 PDT 2012 armv71 GNU/Linux, Asus Nexus 7 Android Tablet
Fingerprint Linux 3.1
Class Asus | Linux | 3.X | general purpose
CPE cpe:/o:asus:linux_kernel:3 auto
(Wow, that's not pretty, since you can find them in your nmap installation in
nmap-os-db so I'm not going to show more.)
I'll pick a simple field
IE() — ICMP echo behaviour (ping is simple, right?)
If you check you'll see that's shared with over 1250 other systems (about two-thirds of which are Linux systems).
From the documentation:
The IE test involves sending two ICMP echo request packets to the target. The first one has the IP DF bit set, a type-of-service (TOS) byte value of zero, a code of nine (even though it should be zero), the sequence number 295, a random IP ID and ICMP request identifier, and 120 bytes of 0x00 for the data payload.
The second ping query is similar, except a TOS of four (IP_TOS_RELIABILITY) is used, the code is zero, 150 bytes of data is sent, and the ICMP request ID and sequence numbers are incremented by one from the previous query values.
This decodes as:
DFI=N no ICMP reply will have DF (don't fragment) set
T=3B-45 TTL will be between 0x3B and 0x45
TG=40 IP initial TTL guess
CD=S ICMP response codes (same as ICMP request probe)
You can find all these codes and details here.
The other fields are contain similarly detailed data for ECN, TCP sequence (SEQ, OPS, WIN, T1) and TCP options (T2-T7).