There are two aspects to this; the first, as you mention, is preventing brute-force attacks.
For this purpose, really any number of tries should do - 3, 5, 20, 2000... with a proper password policy (length+complexity+...) giving a large enough key space, any kind of throttling (X number of tries per hour) will ensure that brute forcing the entire space would take quite a few decades. (Do the math).
Even if - and this should be a requirement - the lockout is only temporary, and after a short period of time it automatically unlocks.
Thus, the number of tries-before-lockout is arbitrary.
However, there is another, more subtle, non-mathematic issue at play here:
It simply does not make sense for a single user to repeatedly put in a wrong password 2000 times in a row.
That is, if you arbitrarily choose 2000, you know long before then that this is NOT a legitimate user. Thus, it really comes down to what makes business sense, and a business-focused risk-analysis trade-off.
I think historically, the trade-off was more slanted towards the risk side - since passwords were shorter and less complex, the difference of 3 or 10 was larger. Also, people had fewer passwords, so they were easier to remember... And, users were more technically savvy in general.
Nowadays, three really doesn't make sense, considering business impact. It's really a question of what makes sense for your app, what types of users, how often they login, etc. I usually recommend to figure out how many failed, legitimate attempts are likely, then double it.
(As @realworldcoder mentioned, PCI arbitrarily chose six, and if you are subject to PCI you don't have much decision here. Otherwise, choose a number that makes sense for you.)