What is the maximum number of bytes for encrypting a plaintext message using RSA that is reasonably secure and also efficient and would AES be better for the same size in bytes? The encryption doesn't have to be public by the way, I'm just wondering if AES is just as good on a short message as it is on a large document. Basically the message or document would be sent encrypted but the key would never be made public. I guess that would also defeat the purpose of RSA but I've read a few times online that RSA is good for short messages and AES is good for long ones.
RSA, as defined by PKCS#1, encrypts "messages" of limited size. With the commonly used "v1.5 padding" and a 2048-bit RSA key, the maximum size of data which can be encrypted with RSA is 245 bytes. No more.
When you "encrypt data with RSA", in practice, you are actually encrypting a random symmetric key with RSA, and then encrypt the data with a symmetric encryption algorithm, which is not limited in size. This is how it works in SSL, S/MIME, OpenPGP... Regularly, some people suggest doing "RSA only" by splitting the input message into 245-byte chunks and encrypting each of them more or less separately. This is a bad idea because:
When encrypting data with a symmetric block cipher, which uses blocks of n bits, some security concerns begin to appear when the amount of data encrypted with a single key comes close to 2n/2 blocks, i.e. n*2n/2 bits. With AES, n = 128 (AES-128, AES-192 and AES-256 all use 128-bit blocks). This means a limit of more than 250 millions of terabytes, which is sufficiently large not to be a problem. That's precisely why AES was defined with 128-bit blocks, instead of the more common (at that time) 64-bit blocks: so that data size is practically unlimited.
Comparing the two directly is a little like comparing a tractor to a train - they're both vehicles but have completely different function and construction.
RSA is an asymmetric cipher. It is ideal for secure exchange of messages across an untrusted network, because the public key can be known by everyone - a message encrypted with the public key can only be decrypted by the private key. As such, if two parties know each other's public keys, they can exchange messages securely. This means that no secret information has to be transmitted - as long as authenticity and integrity are maintained you're safe. Thankfully, RSA provides a method of generating signatures on data, which help prove that it is authentic. Given a message signed by a private key, it is possible to verify that signature using the corresponding public key.
As a rule of thumb, you can only encrypt data as large as the RSA key length. So, if you've got a 4096-bit RSA key, you can only encrypt messages up to 4096 bits long. Not only that, but it's incredibly slow. RSA isn't designed as a full-speed data transport cipher.
AES is a symmetric block cipher, and is incredibly fast. The plaintext is split into chunks called blocks, and each block is encrypted in a chain. There are different ways of doing this, but a common one is called Cipher Block Chaining, or CBC for short. This allows for theoretically infinite message sizes. However, symmetric ciphers like AES require a secret key to be exchanged first. Unlike RSA, the shared key must remain unknown to attackers, so you have to provide authenticity, integrity, and secrecy. That's difficult to do directly.
What you'll tend to find is that both schemes are implemented together, such that RSA is used to exchange the key for a block cipher like AES:
This satisfies a few requirements:
In terms of level of security, it doesn't really make much sense to compare RSA and AES. They do different jobs. We currently assume that 128-bit AES keys are safe, and that 2048-bit RSA keys are safe, but this is entirely dependant on individual security requirements. Using 256-bit AES and 4096-bit RSA keys should be more than enough for the next decade, assuming the implementation is sound.
Note that all of this is a simplification, as there are many caveats and details involved, and describing and RSA exchange as "encryption" isn't strictly correct, but all in all this should be a reasonable high-level overview of the way the two types of crypto work.
AES is a symmetric-key algorithm. RSA is asymmetric. While the both are for encryption, they are often used in different ways, so it is difficult to compare them in terms of efficiency or strength, since the purpose for using one versus the other would likely be a greater determinant in which one or class of encryption is used.
You may want to start with this other question: Encryption - should I be using RSA or AES?
One reason that you normally don't see RSA used for larger amounts of data is because of performance - both RSA and AES are secure for large and small amounts of data (when implemented correctly), but RSA is much slower.
In cases where public-key crypto makes sense, but more performance is needed, a hybrid system can be implemented that leverages both.