When I examined a NMAP -sV scan on a host running Exim on port 465, it uses the standard SYN scan to determine if the port is open, and then issues a Client Hello to which the server responds with a Server Hello. There is a Encryted Handshake from NMAP and Application Data from the server but nothing seems to suggest how NMAP determines what is running on the port, which is quite different than how it identifies other services like IMAP or POP3. So, how does it do it?
It's the same as any other service just over SSL I suspect. So you might just see the encrypted part it's still the same protocol beneath the SSL layer.
Nmap is "smart enough" to recognize SSL encapsulation. In this case, it marks the port as having an SSL tunnel, then on subsequent rounds of version detection it establishes the tunnel, then sends the same probes that would be sent to the plaintext service.
You could see this in action by using
stunnel to convert some other service to SSL-tunneled, then scan it with Nmap while running a packet sniffer on the loopback adapter of the target machine. You would see that the traffic looks the same as Nmap probing an unencrypted service.