This is a really good question and I would like to see more good answers. Here are the ones that I am aware of:
- Hardcode the password(s) in the application.
- Keep the password(s) in a file.
- Store the password in an external system and retrieve it at startup.
Of these the first is the most common and often the password is 'password' or 'changeit' etc. The second option might seem better at first glance but it's hard to explain how exactly. The last option is probably the only one that has a hope of doing anything meaningful. I know of vendor solutions that do this kind of thing and claim to improve security.
The fundamental issue here is that if an attacker is on the host with enough access to see the application space, there's really nothing you can do to keep them from getting this password. At some point it must be in memory in the clear.
As I see it, the password on the keystore protects it mainly in situation that the keystore itself has been retrieved from the machine without actually having access to that machine directly. With that file, an attacker can hammer it at will to get the secrets out of it. If you don't have a strong password on it, game over. If the attacker can pull that file, then they likely can pull other files that the password could be stored in. This puts options 1 and 2 in doubt.
As far as the 3rd option goes, the problem now moves to how you authenticate the clients in order to retrieve the password. I know there are a number of schemes for doing this such as depending on the host/network and using signatures on the binaries. You could have an entire discussion just on how to do this and what level of protection it brings.