Java HostnameVerifier defines an interface for hostname verification. Which specification does it come from? TLS or HTTPS or X509?
The HostnameVerifier interface is specific for Java, i.e. the interface itself does not follow any standards except the specification of Java. The actual implementation of the hostname verification should though follow established standards.
How a hostname verification should be done when checking the server certificate of a TLS connection is mainly defined in RFC 6125 for various TLS based protocols in general and specifically for HTTPS in the older RFC 2818. While there are in theory differences between various TLS using protocols (HTTPS, IMAPS, POP3S, LDAPS...) specifically regarding the use of wildcards most implementations ignore these differences and just implement the way the verification is done in HTTPS.
Apart from RFC 6125 also the CA/Browser forum requirements are relevant. But these are usually only implement fully in the browsers, most programming language don't care about things like Public Suffix lists and their relation to wildcard checking.