Most other methods mentioned here are easily defeated and/or significantly reduce performance.
That is why web application firewalls (WAF) don't rely on them.
2 reliable methods are:
- Analyze the packet sequence in the IP header: Most TCP/IP stacks
number packets sequentially, so if the number is random, then the
client is probably behind NAT (a shared Internet connection with
other clients). There is some RAM overhead to remember the previous packet ID, but it's comparable to a session cookie. This is harder to spoof since the attacker needs to rewrite the packet at a low level, but not impossible, so it should be combined with rate limiting: allow NATted public IPs to have more hits/second.
- Session cookies with cryptographic authentication:
For persistent attacks, you should ask your ISP to block/blackhole the route upstream.
Why not use the other solutions?
'User-Agent:' is very trivial to spoof:
wget -O index.html --user-agent="My Fake Browser"
Session cookies, 'X-Forwarded-For:' HTTP header, and other headers are also trivial to steal/spoof. Google 'Firesheep' or 'Cookies Manager+' or 'Modify Headers plugin' or 'LiveHeaders plugin' etc. for proof.
Rate limiting is not enough alone, either, because a stealth attack will randomize or increase wait time between requests:
wget --limit-rate=10 http://example.com/index.php
Brute force is usually not your only problem. https://www.owasp.org/index.php/Top_10_2013-Top_10 Coding and testing effective protection takes time, too. To save you time and/or CPU cycles on your web servers -- it's multiplied waste if you have a server farm -- your web host should offer a front-end WAF with this configured for you. That is the best solution -- don't do it server-side. Do it upstream.