I know encrypting something with my private key is used for signing. To prove that the message is indeed coming from me.
But what about if I sign something with my public key? That would mean that only I could decrypt it with my private key.
Suppose I did this and I sent over my ciphertext (encrypted with my public key) over to somebody. Will he be able to decrypt it?
There is a common misconception that signing a message is the same as encrypting the message with the private key. This notion is fundamentally incorrect, as pointed out by Thomas Pornin at If the public key can't be used for decrypting something encrypted by the private key, then how do digital signatures work?. As Pornin explains, encryption/decryption and signing/verification are in fact two different algorithms. A private key can be used to decrypt a message that was encrypted using the correspoding public key, or to sign a message; but a private key cannot be used to encrypt a message.
By the same logic, the notion of signing a message with the public key is also fundamentally incorrect. A public key can be used to encrypt a message, or to verify a signature made on a message using the corresponding private key; but a public key cannot be used to sign a message.
There are lots of mixing terminology here. In short RSA Signing is Not RSA Decryption by Cornell CS.
RSA is a trapdoor permutation, unfortunately, that can be both used for encryption and signature. This makes a common confusion.
First of all, although, RSA can be used for encryption, we don't. We prefer hybrid-encryption where a public key cryptosystem is used for key exchange and the key used in the symmetric algorithm. DHKE-AES AES and RSA-KEM AES are examples.
If one really wants to send a message with RSA encryption, they should forget to use the textbook RSA, which doesn't use a padding mechanism to be secure. PKCS#1 v1.5 and OAEP padding can be used for RSA encryption. The latter is preferable since the former is hard to implement correctly that caused many attacks.
If you want to sign a message, you should use the Probabilistic Signature Scheme (PSS). And when signing we don't' sign the message, we sign the hash of the message. This is necessary since the message can be very long and for the security proof.
What happens when I encrypt something with my RSA Public Key?
If you encrypt it with the public key without padding, the cube-root attack works if the public key is 3. Now forget encryption without padding. With correct padding, you have sent the message yourself, nothing more.
I know encrypting something with my private key is used for signing. To prove that the message is indeed coming from me.
But what about if I sign something with my public key? That would mean that only I could decrypt it with my private key.
The public keys are small and assumed to be known if you really use it for signature, this means there is a digital signature forgery. An attacker takes your public key (e,n) and produce a signature forgery.
The correct terminology is not decryption it is the verification of the signature. For signatures, we have sign and verify functions.
Suppose I did this and I sent over my ciphertext (encrypted with my public key) to somebody. Will he be able to decrypt it?
For signatures, the decryption is not the operation. The verification and forgery are the operations. If you use the public key then they will make forgeries.
Final note: Although RSA enables encryption and digital signatures, we don't use the same key for the different operations. You need two different sets for this in RSA.
For the curious reader here the Dan Boneh's article on the RSA attacks.
I’m going to assume that RSA has the security properties that we hope it does. In particular, I’ll assume that, if someone knows a message p, a public key pubKey, and the value of enc(p,pubKey), then it will be hard for them to calculate the corresponding private key, priKey. And, even if they have lots and lots of plain text / cipher text pairs, your private key still will be hard to compute. Hard to compute here means they’d either need a lot of time or a lot of hardware or both. (There is no proof of this assumption, but it seems to be true that know one knows of a way to break RSA when enough bits are used.)
Suppose you encrypt (rather than sign) a message m with your public key, and send someone the result enc(m, pubKey). And suppose that they know your public key too. From the assumption in the first paragraph, with m in the role of p, your private key will be safe. And even if they somehow know m, it will still be safe.
Signing usually means encrypting a hash of your message with your private key, i.e., you send (m, s) where s = enc(h(m), privKey). Then the recipient with your public key can check that dec(s, pubKey) = h(m), which means that s is enc(h(m), privKey) and so (almost certainly) must have been computed by someone who knows privKey.
(This chain of inference in the previous paragraph actually relies on a property of RSA and h that wasn’t stated in the first paragraph, namely that, if someone knows pubKey, but not privKey, it is hard for them to compute a pair (m,s) such that dec(s, pubKey)=h(m). But this is completely irrelevant because the question is about signing with the public key.)
If you mistakenly sign with your public key, then you send (m, enc(h(m), pubKey)). Now the recipient or eavesdropper will know h(m), enc(h(m), pubKey) and, presumably, pubKey. This is the same as the situation described in the first paragraph with h(m) playing the role of p. So again your private key is safe. (The recipient also knows m, but knowing m shouldn’t be of any help unless the message says something like “my private key is ... .“)
