Originally it's to do with the difficulty of a brute force attack on the password.
Most websites are concerned about the possibility that some attacker might get hold of a file containing everyone's hashed passwords, and conduct an offline brute force attack using that. A properly set up attacker might be able to make millions of guesses per second (exact rate depending on whether the passwords are hashed using a suitable algorithm, and just how much silicon the attacker can bring to bear on the problem). So even 8 characters arguably is not sufficient.
Banks (for various reasons to do with their different security models) don't think they're likely to lose files containing PINs or their hashes without noticing, or anyway they're no more concerned than they are with losing millions in any other form of bank robbery. If you want to make a brute force attack on someone's PIN then (leaving aside home Chip and PIN readers), you have to put their card in an ATM or other device connected to the banking system, and type in a number. It's slow, and the machine eats the card after 3 wrong guesses.
Some websites use a similar lock-out to prevent online password-guessing attacks, but the main concern driving the need for password strength is the loss of password hashes. The main concern driving the (lack of) need for PIN strength is use of the physical card (or a clone of it, when using magnetic stripe technology).
Note that there is a still a non-trivial flaw in the simple version of the model I've described. If you steal 10,000 credit cards and make 3 guesses at each 4 digit PIN then you'll expect to get 3 right. Naturally though, a single ATM will notice that something is amiss if it has to eat 100 cards in a row, so I suspect/hope the cops would be on their way before then. Guessing card PINs is risky for the attacker.
In general, banks also pay closer attention to suspect card transactions than websites do to suspect logins. Some websites will try to notice and take extra security steps if they notice a login from a suspect location, adding extra security behind the password. But all card payment systems try to do this. Not that they always succeed.
I don't know what effect home Chip and PIN readers have on this. I just used mine to confirm that my PIN is correct, without any communication to the bank. It might be as simple as the Chip being smart enough to lock itself down after sufficient incorrect guesses. This would still be subject to the 10,000 stolen cards attack. You'd burn 99.97% of the chips, but those could still be used for card not present fraud and the other 0.03% would be good for fraud requiring the PIN. Naturally I'm not about to test that theory with my own card ;-)
Attackers with the ability to steal physical cards on that scale probably aren't messing about guessing PINs anyway. It's simply not the most efficient way to extract money from stolen or cloned cards.
In short, yes there is some risk in using short passwords that they might be guessed. But compared to websites guessing is much harder for attackers, banks defend in depth against card fraud, and they also have higher costs associated with someone forgetting a PIN than websites do with passwords. So they choose a different trade-off.