There are a lot of techniques. Some of them are:
IP TTL values + TCP Window size
ARP / NDP / SEND
SSH / SSL / TLS
Here is a great article about it. Don't forget to read Nmap OS Detection.
To expand upon my comment:
Since popular ransomware like Petya after a reboot replace the MBR (Master Boot Record/ GPT (GUID Partition Table) with malicious code and only then encrypts the MFT (Master File Table, Windows-specific) it would a viable way to go (please keep in mind that not every ransomware encrypts the MFT).
To protect yourself against ...
Detection? Don't count on it.
Watching for changed files may not help. Some malware families reads the file in memory, encrypts it and writes a new file, and deletes the original. This will not trip a "lots of modified files" trigger.
Other will encrypt files in small batches, so there are no spike on disk activity or CPU usage and the user will not ...
The best mitigation against ransomware that has gotten onto your network isn't to stop it as quickly as possible, it's to already have backups.
The first step is to identify the vector and contain all of the potential malware, not just the ransomware.
Until you can rule out additional payloads, you do not want your users on the network where they can ...