To know the strength of your password, don't look at its length, but at its potential. A password is as good as it is random, i.e. as good as the number of other values it could have assumed.
A simple case is when the password was generated as a sequence of random letters, each letter chosen uniformly and independently of the other letters. Since there are 26 letters, a 20-letter password implies 2620 = 19928148895209409152340197376 combinations. An attacker trying to guess the password will need, on average, to try half of them before hitting the right one. The accumulated cost would be tremendous, so the attacker would have very little chance of breaking through it. This number of possible, equiprobable combinations is often called entropy (not exactly the same notion as in physics, but close enough) and traditionally expressed in bits: you have an entropy of n bits if the number of equiprobable combinations is close to 2n. With 20 random letters, this means about 94 bits of entropy.
Moreover, TrueCrypt uses a good password hashing function, namely PBKDF2 with a random salt and 1000 (or 2000) iterations. Password hashing is a subtle matter, and TrueCrypt does it correctly.
To make the story short, if your password entropy is anywhere beyond 70 bits, your security is fine -- meaning that the cost of brute-forcing the password would far exceed the cost of simply obtaining the data through other means, such as abducting you and your family, or simply buying you out. We are talking about several (many) billions of dollars here... So, with random letters (all lowercase, let's make it simple to remember), 15 letters are enough. If you add uppercase letters and digits in the mix, 12 letters suffice, but the password may be harder to remember, and harder to type, especially on tablets/smartphones.
A crucial point of password generation is the need for good, impartial randomness. If you chose the password in your head, then it is not random. Witticism is not randomness; quite the opposite indeed. It is hard to estimate how much random can be a human-generated password, but it is often observed that such passwords are assumed by their owners to be a lot more random than is actually the case.
In the example above, I used letters but the calculation easily expands to any set of "symbols", even words. See this answer for details on a famous example.
The key files serve two purposes:
- To provide some extra entropy to a weak password.
- To provide some extra entropy which does not come in through the same path as the password.
Indeed, a poor password can be brute-forced if it does not have enough entropy, but even good passwords can be preyed upon through various methods: keyloggers, well-placed security cameras, sleep-talkers... Some of these methods will grab the password but not the key file, so the key file may save your skin in some situations.
The key file adds some security only insofar as the attacker does not know the file contents. However, it is a file. It is written on physical media. It is made available to the machine itself upon disk unlocking. We must assume that someone who steals the machine will also have a copy of all the files which could serve as key files (it depends on where you store them, but they are probably close to the machine since you need them to boot it). So the secrecy of the key file is not about the file contents, but about which file is used. For instance, if you have 100 possible files, then entropy is multiplied by 100 (in "bits" notation, this means about 6.6 extra bits). On the other hand, you have to remember which file was used, so that's some extra effort beyond the one needed for the password itself.
I note that key file processing in TrueCrypt is singularly homemade and weird; my cryptographer's senses are quite alarmed at this assembly of CRC-32 and haphazard scrambling. It would have been much simpler and better to simply hash each file, then hash the concatenation of the hashes and the password itself.
Also, the scenarios where key files have any value are rather restricted: these are scenarios where the attacker could grab the password, but not the key file, or not the information about which key file was used. For instance, a camera which has a view on the keyboard may also see your screen and easily obtain that information.
Therefore, I deem key files "not worth it" and I find that using the brain memory capacity to remember the key file to use is better employed at remembering a stronger (more random) password.