There are many ways to do this.
I would encrypt the JSON at a entire file level rather than encrypting content within it. The alternative is complicated and gives no meaningful security advantage. I recommend using an AEAD cipher (my present recommendation is AES-256 with GCM: aka AES/GCM/NoPadding with 256-bit keys). To use GCM you need to be on Java 8 or later. To use AES-256 with an Oracle JRE you will need the Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files installed as documented.
You should make sure to set the authentication tag to include information about location and use (the true filename is a good start) to make it hard to rename or substitute other versions (a form of at-rest replay attack).
However, encrypting the JSON content isn't a complete solution. It just moves the ball a bit. If the attacker still has plaintext access to the private key then it won't be effective.
To protect the private key plaintext you can use a KMS (Key Management Service) on a separate system. That keeps the private key out of memory access on the machine with the encrypted JSON file. You submit the encrypted file content to the KMS to be decrypted on load or vice versa on store.
With those in place you can effectively protect the file from offline attacks (ex: disk image analysis, machine image analysis, backup theft, media theft). Without further defenses, the content of the file is still subject to online attacks of the memory of the machine itself.