There is a sense in which you can define the strength of a particular encryption algorithm¹: roughly speaking, the strength is the number of attempts that need to be made in order to break the encryption. More precisely, the strength is the amount of computation that needs to be done to find the secret. Ideally, the strength of an algorithm is the number of brute-force attempts that need to be made (weighed by the complexity of each attempt, or reduced if some kind of parallelization allows for multiple attempts to share some of the work); as attacks on the algorithm improve, the actual strength goes down.
It's important to realize that “particular encryption algorithm” includes considering a specific key size. That is, you're not pitching RSA against AES, but 1024-bit RSA (with a specific padding mode) with AES-256 (with a specific chaining mode, IV, etc.). In that sense, you can ask: if I have a copy of my data encrypted with algorithm A with given values of parameters P and Q (in particular the key size), and a copy encrypted with algorithm B with parameters P and R, then which of (A,Pval₁,Qval₁) and (B,Pval₂,Rval₂) is likely to be cracked first?
In practice, many protocols involve the use of multiple cryptographic primitives. Different primitives have different possible uses, and even when several primitives can serve a given function, there can be one that's better suited than others. When choosing a cryptographic primitive for a given purpose, the decision process goes somewhat like this:
- What algorithms can do the job? → I can use A or B or C.
- What strength to I need? → I want 2N operations, so I need key size LA for primitive A, LB for primitive B, LC for primitive C.
- Given my constraints (brute speed, latency, memory efficiency, …), which of these (LA-bit A or LB-bit B or LC-bit C) is best?
For example, let's say your requirement is a protocol for exchanging data with a party you don't trust. Then symmetric cryptography cannot do the job on its own: you need some way to share the key. Asymmetric cryptography such as RSA can do the job, if you let the parties exchange public keys in advance. (This is not the only possibility but I won't go into details here.) So you can decide on whatever RSA key length has the right strength for your application. However RSA is slow and cumbersome (for example there aren't standard protocols to apply RSA encryption to a stream — mainly because no one has bothered because they'd be so slow). Many common protocols involving public-key cryptography use it only to exchange a limited-duration secret: a session key for some symmetric cryptography algorithm. This is known as hybrid encryption. Again, you choose the length of the session key according to the desired strength. In this scenario, the two primitives involved tend to have the same strength.
The same notion applies to other uses of cryptography, such as signing or hashing.