On a "forgot my password" page, is appropriate when user doesn't have an account to display the message "this account doesn't exist"? Or in the interest of security, should I display a success message ("you received an email with a reset password process") even if no email was sent?
The answer to this depends on the nature of the site. I actually think the UX benefits of telling them if the information provided was correct or not, but only if it doesn't impact security.
There are plenty of sites where you are able to enumerate usernames is multiple ways, and they are that way by design. For example, if you wanted to enumerate Twitter usernames, you could easily do so by seeing if the user's page were available at http://www.twitter.com/username. Even if that wasn't possible, Twitter provides a nice little API for accessing lists of users.
However, this obviously doesn't work for a site that is even slightly sensitive in nature. If your site falls in this category and there is no other way to enumerate usernames/emails, then follow your instinct and give the generic message that you will send an email.
From a security and privacy standpoint, it's best to say:
"If your account was located, an email will be sent with instructions for resetting your password."
However, that's obviously not helpful for someone who isn't sure what email or username they registered with.
But I'd sacrifice a little user convenience to provide additional security and privacy. So, in other words, do not say something like "that account doesn't exist."
From a usability point of view, it is horrible to show a generic message no matter if the username/email entered was found or not. It is extremely annoying if you don't know for sure which username/email you used.
However, there is an easy way to keep it user-friendly while still not allowing users to enumerate accounts:
Provide fields for both username and email and require just one of them. If an email address was entered, send an email to that address with the necessary information. If a username was entered, send an email to the address you have stored for that username.
While this may allow people to trigger reset-password emails for other users by accident it's not a big issue - after all, the email should just contain a link to a form allowing to reset a password and not immediately change the password to a random one.
Genuine users of the application may find it confusing if you give out the message as "you received an email with a reset password process"
In my opinion, to prevent user enumeration, you can add a security question as well.
Like adding an option to check the date of birth, or a detail that some random attacker may find hard to answer.
For mismatching combinations, you can make sure an error message is displayed as "mismatching combination". That will confuse the attacker. But a genuine user will find his way in using next try.
To prevent a brute force you can also enable CAPTCHA, and a balanced temporary lock down for a small amount of time based on number of false tries.