How do you properly implement defenses against brute forcing? Is it best to store how many times someone tried to log in and block them after X attempts? And how would "someone" be identified? With the session? An IP?
Account lockout is one option, where the account is locked either permanently or for a period of time after x unsuccessful logins. The problems here are Denial of Service risks if an attacker attempts to guess passwords and also the administrative overhead of resetting accounts (assuming an automated reset isn't available)
Another option is to introduce CAPTCHAs into the login process. This is intended to limit automated attacks, and stop the denial of service from account lockouts. The problems with it can be accessibility requirements and automated CAPTCHA cracking. From what I've seen some CAPTCHAs are easier for software to crack than others.
A third option is to introduce a progressive delay into the login process, so for each wrong log in you cause the process to become ever slower. This has a limited effect on real users but makes automated brute forcing much harder.
In terms of the second part of your question about identifying the "someone" doing the attack, it depends on what they're doing. If they're just attacking a single user, then the username is the identifier and actions are tied to that account. If they're brute forcing across a wide range of users then source IP (perhaps combined with browser agent) is an option. It's not perfect (eg, use of botnets, agent spoofing) but is probably the only information you'd have to go on.
The answer to denying brute force is to deny a large number of attempts, where large is defined by your key space and risk acceptance.
Each situation is going to have different solutions.
All of the solutions also have trade-offs like everything else in life.
As I suggest, you mean bruteforcing login/password in web-application.
No one bruteforces manually. So, if there are doubts that this is human who tries to authenticate, show Captcha, say, from the 3-d failed login attempt. Further you can put timeout between login attempts - but this will not work for distributed attacks from different IP's. Next, there are two types of bruteforcing - blind, just trying random login/password, and attack against some user. For the last attack it is effective to implement such feature like blocking account for some time and notifying user by e-mail. For the blind bruteforce mitigation can be harder to implement, especially, if the site is popular. In this case you can track such user information like IP/Country/Browser/etc, and figuring out combination of this, you can make assumptions of whether this is valid user. As and addendum to this, you can play with long-living cookies - once the user successfully logged in, save cookie, and later check for it.
Without web-application type solution it is possible to implement server-level solutions like Apache's mod_evasive. Or even write own web-server module, specifically for such purposes.
Sure, there are many other ways how to harden bruteforce attack mitigation, but often they comes out to be noisy, rather useless and nasty. Try to keep with KISS.
The important thing is how do you identify the 'someone'?
You can't do it by IP address - many users may appear to have same client address (this is going to become even more frequent with the continuing problem of IPV4 address starvation).
One user may appear to connect from multiple client addresses - e.g. using AOL's load-balanced proxies, or less legitimately, via tor.
Anything else is data supplied by the client - so they can potentially modify any cookies you set, the user-agent string etc.
The only practical approach is apply heuristic scoring to the request to try to match it against previous attempts - but its still going to be impossible to differentiate between sophisticated attacks and legitimate users.
Set a score threshold at which you consider the client to be trying to brute-force your site, say -20.
Requiring a current valid session referenced by a session cookie (which is not set in the page prompting for a username/password) is a start - if the cookie is not presented with authentication details then redirect to a seperate page which drops the cookie and initializes the session with a score of -10, and a score of (say) -2 to the client IP address. You could use a dynamic scoring mechanism when you start seeing multiple valid users from the same address. Similarly you could maintain dynamic scoring by user-agent and by the username.
When cookie+auth are presented for a non-existent user, add a score of -5 to the session, -1 to the client address, -5 to the username.
When cookie+auth are presented with an invalid password, add a score of -3 to the session, -1 to the client address, -4 to the username.
When cookie+auth+valid password add +4 from the score for the client address, +3 to the username, +2 to the user-agent.
NB you also need to set a decay to allow any score held against client address/user-agent/username to recover, say +2/hour
You need to think about what happens when the score exceeds the threshold. Have you provided a mechansim for anyone to block access to your site?
If you've blocked access with a high score for a prticular username, send out an email with a reset url to the registered user for that account so they can reset the score held for their username and reduce the score held for their user-agent/address.
Store the long ips of previous logins and login attempts.
After N failed attempts put them against captcha.
Block quickly addresses that have no successful logins in their history but a number of attempts.
Block quickly addresses that have simultaneous source address or inhuman speed / staminarapid attempts.
Also none of these answers take into account the method of brute forcing where an attacker uses one password and tries it against all accounts, a reversal of the usual process, and one which isn't generally protected against.
The control would of course be to have a process monitoring for multiple attempts against different accounts with the same password, but again very tricky to block even if they come from a single address, as do you block that IP? It could be used by valid users as well, or do you just drop logins with that password - with the risk of locking out a valid user with that password?
And of course if an attacker runs this sort of thing from a distributed network, or over a long timeframe, how do you correlate the incidents into an attack profile?
General industry mechanisms include lockout
First, it makes sense to understand which scenarios/attacks you're intending to handle:
(See? Not so simple is it...) And there are different solutions for each part of these...
If it is a business website use certificate plus password authentication. Without a valid certificate they never get to the point of trying to guess the password. You could also use something like RSA SecurID in place of the certificate.