In the RSA cryptosystem, encryption, decryption, signing and signature verification are all done with the same mathematical operation: modular exponentiation.
Also, mathematically, there is no difference in RSA between the private key and the public key; they're just two pairs of numbers (e, n) and (d, n), where the shared modulus n is a product of two (or sometimes more) large primes, and e and d are chosen so that xed ≡ x (mod n) for all x.
Conveniently, it turns out that choosing such numbers e and d is easy to do if you know the prime factors of n, but very hard if you don't (even if you know one of e and d already). So you can pick two random large primes, multiply them together to get n, choose suitable e and d, and then publish n and one of e or d (conventionally, we call the public one e and the other one d) while being pretty confident that nobody else can figure out which number d corresponds to your n and e.
(The math even allows you to fix one of e and d to be (almost) any number you like, and then compute the other one based on it and the prime factors of n. For example, you can always pick e = 3 or e = 65537 = 216+1 to make calculating xe mod n a little easier. Of course, if you do that, you'll want to make the constant number the public one, because if you and everyone else always used the same secret number d, it wouldn't really be much of a secret, would it?)
It then also turns out that, while it's easy for anybody to compute y = xe mod n for any x, and for you (since you also know d) to compute x = yd mod n, it's very hard for anyone who doesn't know d to figure out which (sufficiently randomly chosen) x corresponds to any given value of y. And there are at least two convenient practical applications for this property:
You can let anyone "encrypt" a number x by computing y = xe mod n, so that only you can easily "decrypt" the resulting y back into x = yd mod n.
You can "sign" any number y by computing the "signature" x = yd mod n and publishing it, and anybody can verify that your signature x matches the original number y = xe mod n, but nobody except you can easily come up with a signature x that would match a specific number y of their choosing.
Of course, in practice, there are some limitations to this. For example, when using RSA for encryption, it's important for the number x being encrypted to always be unpredictable; if somebody can guess a likely value for x, they can just compute xe mod n and see if it matches the actual encrypted number y.
And similarly, when using RSA for signing, it's always possible for anybody to just pick a fake "signature" x and calculate a (essentially random) number y = xe mod n that matches the signature x; thus, it's important to restrict the range of valid numbers y to only a very small subset of the full range from 1 to n−1, or to otherwise ensure that such signatures of (pseudo)random numbers are useless as practical forgeries.
The usual way in which these practical issues are solved is by padding the input numbers (x for encryption, y for signing) in ways that make them unpredictable and/or allow the correctness of the padding to be verified (in such a way that randomly chosen numbers are vanishingly unlikely to pass the check). In practice, different padding schemes are generally used for RSA encryption and for RSA signing, since the security requirements for these two distinct uses are somewhat different. Thus, while the modular exponentiation part is essentially the same for all four RSA operations (encryption, decryption, signing and signature verification), the additional (un)padding operations required to actually make the system practically secure are different.
Anyway, the point of all this is that your original question is based on a false dichotomy.
Insofar as it's meaningful to describe RSA signing as being equivalent to "encryption with the private key" (i.e. insofar as you ignore the padding and just focus on the modular exponentiation), it's also just as meaningful to say that it's equivalent to "decryption with the private key", or even "verification with the private key", since all of these are, mathematically, really just different names for modular exponentiation. Except, of course, that in practice all those operations also involve some (un)padding, which makes all of them different.
And of course, the reason why you always use the private key for signing (and for decryption) is that if you used the public key for it then, since it's public, anyone else could do it too. And that would defeat the point of using crypto in the first place.