First of all: it depends. There are cryptographic algorithms that can be used both ways like RSA, but also others that can only be used for encryption or signing (in OpenPGP, those usually occur in pairs, like DSA for signing together with an ElGamal subkey for encryption).
Is it possible to encrypt a message with my private key and have the recipient decrypt it with my public key?
In the case of RSA, signing a document technically is equivalent to encrypting the hash sum of the document with your private key; if a third party can decrypt it with your public key, the document must have been encrypted by your private key (and given you did not hand that out, it was encrypted by you).
Other encryption algorithms like ElGamal cannot be used like this, on the other hand signing algorithms lika DSA cannot be used for encryption.
I know that this would provide no real security since my public key is widely known.
It would of course not provide confidentiality, but also authorship can be considered security!
It would also be a very impractical way of establishing that I authored the message since PGP already supports signing.
Considering OpenPGP specifically: If you create an RSA primary key that has both signing and encryption capabilities set, you actually use the same public/private key pair for both encryption and signatures as you described. In GnuPG, you will have to provide the
--expert flag while creating the key. Doing so is discouraged, though; splitting duties might prevent some (yet unkown) attacks at least for some usage of your key/key set.