OpenPGP Includes Key Management
From the introduction to RFC 4880, OpenPGP (highlighting added be me):
OpenPGP software uses a combination of strong public-key and
symmetric cryptography to provide security services for electronic
communications and data storage. These services include
confidentiality, key management, authentication, and digital
With other words, OpenPGP is not just about sending encrypted messages, but provides a much larger feature set around key management. Key management means the possibility of looking up your key on the key servers (but they're not authenticated there!), but also users of OpenPGP issuing certifications in-between their OpenPGP keys to assure authenticity (for example, read up on keysigning parties). All this requires keys to have names attached.
"Anonymous" OpenPGP Keys
Why am I asked for my name or email?
You're not forced to do so (although some implementations of OpenPGP do): OpenPGP keys may exist without user IDs, and all dates can be set up rather arbitrarily (although a key creation timestamp must be provided, but you could just provide a timestamp of 0 equalling 1970-01-01).
Consider keys would not have any user information attached: you'd have to exchange the whole key composed by thousands of random bytes manually, and somehow connect it to an identity if you want to use it.
Another suspicious thing was after what time the key should expire. Is the information about expiration date contained within the key (I don't think it is) ? So is the key stored on a server somewhere or what? How does it work?
To expiry date is not required, you can have them be valid forever. Actually, the expiry date does not really add up to security at all, it might be reasonable nonetheless.
You have to distinguish between the actual public/private key pair (eg., the primes used for RSA) and an OpenPGP key. An OpenPGP key is composed by the numbers for the crypto works, but also additional information like user IDs, certifications and settings like the expiry date. You can easily have a look at all the information in an OpenPGP key by running
gpg --export [key-id] | gpg --list-packets
(some reading in RFC 4880 linked above might be required to get a reasonable understanding of the output).
OpenPGP is Distributed
The reason I ask is that this whole PGP thing seems more centralized than I thought.
There is no central instance in the OpenPGP environment. Keys are exchanged through a decentralized key server network, trust is validated through the OpenPGP web of trust instead of a hierarchical PKI system like known from X.509 (used for S/MIME, TLS, ...).
There are some rather central authorities in the OpenPGP community: certificate authorities like CAcert, the German Heise Verlag and Governikus issuing certifications for holders of the new German digital identity card exist, but you don't need to use (or trust) them. There is some rather central coordination in the key server network (the SKS pool), but you can also choose an arbitrary server, not use key servers at all or even host your own!
OpenPGP is an Open Standard
Why can't the PGP software just have a "Generate a new key" button which generates both the public and private key for me to copy and use (+ of course an option to encrypt/decrypt text)?
OpenPGP is an open standard. If somebody wishes software that does exactly this (and as I already mentioned, the OpenPGP standard allows rather "plain" keys without a lot of meta information), you're welcome to do so. The reason such software does not exist (or is not widely known) is that people don't see a good reason to write or have such software.
If you just want to use RSA and AES without any key management and the advanced OpenPGP capabilities, you're probably better of just using more primitive standards.