There are two aspects to this.
The first, as you mention, is preventing brute-force attacks.
For this purpose, really any number of tries should do - 3, 5, 20, 2000... with a proper password policy (length+complexity+...) giving a large enough keyspace, any kind of throttling (X number of tries per hour) will ensure that bruteforcing the entire space would take quite a few decades. (Do the math).
Even if - and this should be a requirement - the lockout is only temporary, and after a short period of time it automatically unlocks.
Thus, the number of tries-before-lockout is arbitrary.
However, there is another, more subtle, non-mathematic issue at play here:
It simply does not make sense for a single user to repeatedly put in a wrong password 2000 times in a row.
That is, if you arbitrarily choose 2000, you know long before then that this is NOT a legitimate user. Thus, it really comes down to what makes business sense, and a business-focused risk-analysis tradeoff.
I think historically, the tradeoff was more slanted towards the risk side - since passwords were shorter and less complex, the difference of 3 or 10 was larger. Also, people had fewer passwords, so they were easier to remember... And, users were more technically savvy in general.
Nowadays, 3 really doesnt make sense, considering business impact. It's really a question of what makes sense for your app, what types of users, how often they login, etc. I usually recommend to figure out how many failed, legitimate attempts are likely, then double it.
(Btw, as @realworldcoder mentioned, PCI artbitrarily chose 6, and if you are subject to PCI you dont have much decision here. But otherwise, choose a number that makes sense for you.)