As @whoami mentioned, it does just move the issue to another place. As you mentioned it allows you to move it out of code (if you've hardcoded it), password protect it (if it was originally just a PEM or whatever on disk), and ACL it to the requisite permissions.
The problem is that you do need to know the key to unlock it, and that of course requires protection of some sort, and well, you get turtles all the way down.
What you're basically doing is moving the bar for the attackers so they have to put in more effort to break the thing. At some point they just say screw it, and move on. You can make it harder by introducing complexity to key generation, but if attackers know how this works they can do it themselves easily enough, or worse they can just steal the file and try brute forcing the key since its not particularly random. In some cases the OS does provide a mechanism of protecting the random keys, but in this case you'd need something that is exposed to the JVM.
More practical solutions involve moving the key to better-protected areas, and that usually means hardware of some sort. In some cases we're talking TPMs, HSMs, or simply smart cards (though they are all really the same thing, but with different features). The basic principle is that the key never leaves the hardware, so its impossible to steal it. A side effect of this is that all crypto usually occurs in the hardware itself, so all you're doing is shoving data in, and getting data out, without needing to worry about the crypto itself.
This then means you need to protect the interface to the hardware so only authorized systems can talk to it. See the turtles yet?
