If I understand you correctly, "no one else" means in your case "not my team" (so persons with access to source code, servers etc.).
So I see only one solution:
- Extract the signing functionality (create a signing service/server).
- Run this in an dedicated environment/server/vm (store the key here).
- Restrict access to this as required.