I would like to add a little bit more to the already outstanding answers.
First it is useful to distinguish among "primitive" (e.g., AES, SHA1), "protocols" (e.g., TLS1.2, DNSSec, etc) and "constructions" (e.g., HMAC, PBKDF2, AES-CBC). What you are asking about is a particular construction. (Note that the distinction between "construction" and "primitive" depends on what you are talking about. If you are looking at how SHA1 works, you will see that it is a construction of more primitive things.)
So I will break your question down into three parts
- Why do we need (complex) constructions?
- How to people devise constructions?
- WTF is up with
key = hex( rsa( plainText + hex (sha1( plainText ) ) ) ) ?
Why do we need constructions?
The classic example of a construction are block-cipher modes. Because a block cipher like AES is a function that takes a key and a block (128 bits) of data and returns 128 bits, every time you encrypt the same block with the same key, you will get the same result; that's what you expect from a function. Because this can lead you to see penguins, we put AES in a construction that makes sure that each block is actually encrypted with a different key or that each block undergoes some transformation before it is encrypted.
Another common construction is HMAC. If Alice and Bob share a secret key, k, and Bob wants to make sure that the messages he gets from Alice really are from Alice they can use k to create a Message Authentication Code (MAC). Now the way that almost every developer naively comes up with for creating a MAC is to send
MAC = SHA1(k, message) along with the message. Alice can create that MAC, and Bob can verify it by performing the same computation on the message and seeing if he gets the same result.
Unfortunately that naive MAC construction is vulnerable. Given how SHA1 (and almost every other cryptographic hash algorithm works prior to SHA3) Mallory can take the message and the MAC and add stuff to the end of the message and construct a new MAC based only on the original MAC and the new material. That is, Mallory can actually use the MAC that Alice sends as a key for creating a new MAC for the extended message. So instead of using the naive MAC construction, we use HMAC, which looks like
HMAC= H(k_o, (H(k_i, message))) where H is some hash algorithm (and k_o and k_i are derived from the k in a specified way).
Someone looking at HMAC might think that it is overly complex, but it is no more complex than it needs to be in order to be able to do its job. So this is why we use often complex constructions.
How do people devise constructions?
The process is as @tom-leek described it. It involves a process of asking explaining what is expected from the construction, why the parts are there, and then asking people to try to break it.
If you want to see this process in action, take a look at the password hashing scheme competition. There is a community that is looking for a successor to PBKDF2, scrypt, bcrypt. So if you look at the mailing list archives you can see this process. One of the motivations for that project is to discourage things like what we find in the third question ...
What is up with that rsa, sha1 construction?
There is no way to know without asking the person who came up with it. I'm guessing that they'd seen standard salting advice that recommends
hash(salt, hash(password)), but don't fully understand what that is for.
RSA in here is particularly odd. First of all, there doesn't seem to be a key in there; so I don't understand that even at the basic level (much less the intent). But assuming that the key is some constant known by the system, I'm guessing that this is a particularly obtuse way of going after the "secret salt" approach to password hashing.
The idea behind a secret salt approach is that even if your password hashes leak, you can make the work factor for someone trying to crack the hashes much harder than the work factor for the legitimate authenticating server because the legitimate server has access to a secret that won't be in the database of password hashes. The secret salt might be available only from a Hardware Security Module (HSM) or such.
But even if that is the intent here, RSA is just not the right way to go about doing this. But as I'm already speculating wildly about the intent, I won't go into further detail.