Assuming that the passwords must consist of printable ASCII characters (of which there are 95), the number of possible 32-character passwords is 9532 ≈ 2 × 1063.
Meanwhile, the number of 31-character passwords is 9531 ≈ 2 × 1061, i.e. about two orders of magnitude less. (In fact, they differ precisely by a factor of 95.) The number of 30-character passwords is even less, 9530 ≈ 2 × 1059, and so on. In fact, the total number of possible passwords of length less than 32 equals:
9531 + 9530 + 9529 + ··· + 952 + 951 + 950
= 9532 × (1/95 + 1/952 + 1/953 + ··· + 1/9531 + 1/9532)
< 9532 × (1/95 + 1/952 + 1/953 + ··· + 1/9531 + 1/9532 + 1/9533 + 1/9534 + ··· )
= 9532 × 1 / (95 − 1)
= 9532 / 94
Thus, the number of possible printable ASCII passwords of exactly 32 characters is 94 times greater than the total number of such passwords of less than 32 characters. Thus, if an attacker is able to crack a 32-character password by brute force, it will take them, on average, one 94-th of the time to crack any shorter password.
Of course, this assumes that your passwords are randomly chosen out of the space of all eligible passwords. Few real-world passwords are such, but as a rough approximation, we can still assume that the amount of work needed to guess a password by brute force grows exponentially with the length of the password — the base of the exponent will just be something smaller than 95.
Also, there's nothing special about the number 32 in the calculation above. Replacing it with any other maximum length will yield exactly the same result: it's always a good idea to make your passwords as long as possible, at least up to the point where the password is long enough to make brute force cracking essentially impossible.
Where is that point? Well, the general consensus among cryptographers is that 264 ≈ 2 × 1019 brute force guesses can be done with enough effort, while 2128 ≈ 3 × 1038 guesses is probably beyond the capabilities of any currently existing or foreseeable attacker (yes, even the NSA), and 2256 ≈ 1 × 1077 guesses is definitely beyond the capabilities of mankind. The effort required to crack a random 32-character ASCII password by brute force, i.e. 9532 ≈ 2 × 1063 guesses, is between the latter two, but definitely closer (on a logarithmic scale) to 2256 than to 2128, both of which are currently regarded as unbreakable. Thus, we can safely say that, if your password is a random ASCII string, 32 (or even 31 or 30) characters is quite plenty enough.