In addition to the other answers, you might want to try to increase the security of your browser. Default levels are a compromise between security and ease of use.
For Firefox, there is a list of SSL related addons that may help. There are also instructions on improving the security of SSL connections.
There is also a web site that helps you understand whether you have the basics right for SSL connections.
Getting SSL connections right is one of the important things as it is your secure connection to your secure web sites.
I would actually recommend also that you put the whole secure setup on a large FLASH USB drive and boot from it when you want to securely go online. That way, you are not accidentally leaving traces behind on your normal partition. Keep the USB drive in the safe when not in use.
By the way, the reason you haven't hit on anybody recommending this way of doing things is simply that it is a pain and few people are that bothered.
If you are keen to have a USB bootable, secure environment, you have some options.
Firstly, you might consider using BSD (see here and here) instead of Linux as this is generally considered more secure by design.
For Linux, there are some articles that explore what you are trying to do. Here for example, here.
You can add some additional security by ensuring that the USB drive is fully encrypted. This can be via a whole disk encryption (or here) or by using secure USB drives, for example the LOK-IT drive.
Be careful though if you travel with these devices since there are plenty of unenlightened corners of the world, including the USA, whose customs officials will assume that you are a bad guy, if not a terrorist, if you are "evil" enough to carry encrypted data with you. Ref: 1, 2, 3, 4, 5, ...