The point of a password is to prevent the key from immediately being compromised once it leaks. It should be considered partially compromised though as the security is reduced from the keysize down to the password complexity. If you have a high complexity password, it may still be an acceptable level of security for your needs, but generally it is advisable to work on replacing the key since security is reduced.
The urgency of the transition depends on your password though. If your password was "password" then you should immediately stop using it and transition, but if it was something reasonably complex, then you should be able to keep using it for a bit and distribute your new public key to your contacts using it. Don't use it for anything you need long term privacy on, but it should be fine for communicating out your new credentials with confidence that you still control the private key for the reasonable future.