Does it matter if a brute force search for a password returns a collision and not the password?
If a password is the only mechanism for accessing a resource, it often does not matter if it is exact or a collision. If you enter "dog" or "cat" and your hashing algorithm was string.length, then either will grant you access.
While this is often the case, it is also a matter of the program/service's implementation details. For instance, a site/service may perform authenticity and runtime validity checks, which assess the password and where the password is coming from. So even if "dog" and "cat" could grant access, if the middleman has a rule that passwords can't begin with "d" then that collision is out.
Additionally, some more secure services sometimes collect metadata (also hashed) on the supplied password (e.g., length, number of bytes, etc.) and inconsistencies in that metadata is sometimes used to flag, deny access, or reset password to accounts.
OP Example
h(3) = 3 mod 17 = 3
Is the problem that the password hashing space (0-16) is much smaller than the space of the allowed password, or is there something else I'm overlooking?
To address your question of the problem; there are many problems with the example; including, simplicity and too many potential collisions. Without any rules of enforcement, the issue isn't about the entry window size. The problem is the hashing algorithm leads to too many collisions; 3 plus multiples of 17 ({3,20,37,54...}
) all result in 3.
The collisions are what’s important because it’s the hash that is used for authentication and verification; if the brute force can enter the 20 or 37 or 54... and still access the account or resource, the original password doesn’t matter.
To circle back to the original question, it is often the case that all that is required is the collision. In fact, this is often how WiFi intrusion works. The hash is found, which allows connection to the password-protected gateway.