This is a type of timing attack, leading to a username enumeration vulnerability.
Whether this is a threat or not depends on the design of your system. If usernames are supposed to be private them it is a concern. Some systems are written in such as way that the enumeration of users fits into the design, such as email providers, as email addresses are generally public and if someone has the email address
[email protected] then they definitely have some sort of email account at
example.com and there is nothing leaked by letting other users know that this exists. The username being public carries more of a risk as an attacker will know which accounts can be targeted. In an email system though, this can be achieved by an attacker sending an email to the address to find out if it bounces so there is little you can do to narrow this attack vector in this case (if you want typos in email addresses to be notified to the sender of course).
Basically the vulnerability is summed up well here:
As an attacker if I can use your login or forgotten password page to narrow my list from 10000 targets to 1000 targets, I will.
Sometimes it is assumed that if a website allows user registration with user chosen usernames or a password reset then user enumeration is always possible. This is not always the case - a user registration system can be combined with a password reset system and allow a user to select a unique email address as their username without revealing to other parties that it is already in use. This is accomplished by always returning the same message to the user on the website, but emailing the link to the next step of the process to the provided address. This ensure that only someone with access to that email address can either register or reset the password. Of course this would be for a medium security system where it is assumed an email MITM or other compromise is not a viable or worthwhile attack vector.
Timing attacks by their very nature will take time for an attacker to execute. So they might want to use this attack on your system in order to narrow their list before attempting a brute force attack on the "slow" login attempts, which will also take time.
If you wish to narrow the attack vector then you should introduce an artificial delay that will make all logins whether they have a valid username or not take around the same amount of time.