In the process of designing a web app that handles public/private key pairs - a Bitcoin web app, specifically - I've had a lot of thought on how to securely handle the storage of the private key, since, using a banking analogy, if someone has access to a user's Bitcoin private key, they have access to all of that user's Bitcoins.
In all scenarios the private key is going to be encrypted in a database that's as secure as possible - no direct Internet access, explicit read/write privileges, etc - but the part that I'm trying to determine and figure out the best solution to is the private key deception. I can do it either on the client side, i.e. in the user's browser, or on the server side.
If I do it in the server side, I could ensure that it's generated and encrypted cryptographically soundly, but if a malicious person gets access to the server, that malicious person will be able to access any decrypted private keys in memory. I could send it to the browser to hold in its storage but then that can raise problems of man-in-the-middle attacks being able to read the private key in transit. All of the decryption/encryption of the private key would be done on the server, including when the private key is first generated.