Mathematically, an RSA key pair is a number n which is the product of two large probable primes p and q, and a pair of numbers (d, e) such that d × e - 1 is a multiple of (p - 1) × (q - 1). The numbers d and e are known as exponents (for a reason we'll see in a minute). There are a few more constraints but they treat d and e symmetrically. The public key is the pair of numbers (n, e) and the private key is the pair of numbers (n, d).
(Private key files often contain more than just n and d, but that's a matter of concrete format, and a matter of private key files usually containing the public key too.)
So in this respect, yes, RSA keys could be swapped.
However, the point of a public/private key pair is that one of the keys remains secret while the other is made public. The number n has to be used on both sides, so it's public. One of d and e — conventionally, the one written d — is kept secret and e becomes public. If d is to be kept secret, it must not be easy to guess.
The heart of the RSA operation is raising a large number (about as many bits as the key size) to the power of d or e. This is an expensive (slow) operation, so it is nice to choose d and e to make it not too slow. This entails making d and e small (or as a second-best having few bits set to 1 in their binary representation). However, if you make d small, you make it guessable by enumeration. So d, the private exponent, has to be large. On the other hand, e can be kept small; for mathematical reasons it has to be odd and cannot be 1, but 3 is a perfectly good choice, and the secondmost common one. For historical reasons, the most common choice of e is 65537, which is 10000000000000001 in binary.
Thus in an RSA key pair as is done in practice for good reasons, given the private key (n, d), one can find the public key (n, e) by guessing possible values of e (is it 65537? If not, is it 3? If not, try a few more small numbers.) For this reason, the public key and the private key cannot be interchanged.
Note that this is a property of RSA. Some other public key algorithms (such as DSA) have public and private keys which live in different mathematical spaces.