When I generate key it has my email address attached to it. Does it mean, when I publish my public PGP key people also see my email attached to it? From what public keys I saw so far it seems to me everybody can see the address, but I am somehow not sure.
Yup, absolutely. PGP was not designed for anonymity -- quite the opposite in fact, it's designed so you can prove that you are who you say you are, and so people you've never met in person can easily look up your key and send you secure email. The idea of "anonymous PGP keys" doesn't really make sense.
Name and email address are required fields when creating a new key, and they get embedded into the key (along with a photo if you want). There is of course nothing stopping you from putting garbage into these fields if you are trying to hide your identity, but the better PGP clients I've used automatically use the
From: fields in the email to decide which keys in your PGP contacts list encrypt for / verify against. Putting in a garbage email address might actually make your key very difficult to use.
For example you can easily find my PGP key by searching my name or email address on the public pgp key servers, this is how the system is designed to work:
pub 4096R/62F715FD 2015-02-24 Mike Ounsworth <firstname.lastname@example.org>
You are not required to publish your key to a key server (but the more you use it, the higher the chance that one of your friends' clients will automatically publish it for you).