Should I really keep my private gpg key secret if it's protected by a passphrase? What if an adversary stole it? And if it didn't have a passphrase, would it be game over for me?

Update:

What to do if my private key is stolen?

Since it's easy to crack a passphrase, does it make any difference if it has or not a passphrase if and only if private key is stolen?