Size and performance don't matter, as a hardware security module (HSM) is defined by its functions to perform cryptographic operations and protection. From Peter Smirnoff on Cryptomathic: Understanding Hardware Security Modules (HSMs):
The hardware security module (HSM) is a special “trusted” network
computer performing a variety of cryptographic operations: key
management, key exchange, encryption etc.
It seems to be obvious that cryptographic operations must be performed
in a trusted environment. When I say trusted, I mean “no viruses, no
malware, no exploit, no unauthorized access.” An HSM is trusted
Is built on top of specialized hardware. The hardware is well-tested
and certified in special laboratories.
Has a security-focused OS.
Has limited access via a network interface that is strictly controlled
by internal rules.
Actively hides and protects cryptographic material.
You can find the same list of the key properties of a HSM e.g. from Doron Gez: What Is a Hardware Security Module. Although YubiHSM is nano form factor, it meets these requirements (YubiHSM 2 Product Overview). While bigger appliances could do these operations faster and store more keys, that only affects their possible use cases.