Your proposed hashing scheme has no real advantage over existing schemes and some potential downsides.
A password hashing scheme must have three key properties:
- it must be one-way: no way to recover the password from the hash, the only way to find the password is to guess it;
- it must be salted: each hash incorporates a unique value so that an attacker can't reuse the same computations to attack many hashes;
- it must be expensive (at the very least, slow): calculating a hash must require significant resources, so that mass attacks require a huge investment in hardware.
Your proposed scheme is one-way — but the first hashing step with MD5 is sufficient for that. (Incidentally, although MD5 is has no known weaknesses for this use case, it's sufficiently broken that it's strongly recommended not to use it at all. But you could just substitute SHA-256 and your question would remain valid, as would my answer.) Performing an RSA operation on top doesn't improve this aspect.
Your proposed scheme is salted, by using a different key for each hash. That's fine. But there's an easier way to do the salting: you essentially take hash(password + salt) instead of hash(password). (The devil is in the details but I'll stick to the high-level summary here.) So we already know how to do this.
Your proposed scheme is not expensive. Expensiveness needs to be tunable, to go to the maximum resource usage that is bareable on the legitimate server that verifies the password, and keep up with technological evolution (e.g. increase the number of iterations from 105 to 106 every 5 years as computers are ~10 times faster). A simple way to do this is to take a fast hash and iterate it, which requires a long sequential computation; this is what PBKDF2 does. More complex schemes try to be optimal on PC hardware and less good on e.g. GPU or FPGA; see Do any security experts recommend bcrypt for password storage?. It is not clear how your proposal could be tuned for expensiveness: use larger RSA keys?
A major potential flaw in your proposal is that the core RSA operation (modular exponentiation) does not have good security properties when used directly, because it is subject to algebraic relations such as xe·ye = (x·y)e. This is why “RSA encryption” is not a simple exponentiation, but instead adds some padding to the data; OAEP is the modern standard. You cannot use OAEP as it is because it's non-deterministic: running it twice on the same input yields different ciphertexts. It is probable that a derandomized version of OAEP would work, or even some simpler scheme given that you're in a very special case where no ciphertext is ever going to be decrypted, which means that many attacks on encryption schemes are not applicable.
Note that you cannot go as far as not adding any padding, not without additional constraints: y = xe mod n absolutely must wrap modulo n, otherwise x can be computed from y simply by taking the eth root in the reals. As specified in your question, e is not constrained, allowing the common choice of e = 3. Since x is a 128-bit number (256-bit if you use SHA-256 instead of MD5) and n is a 2048-bit number, xe would not wrap, so x can be recovered trivially from y. There's still the one-wayness of the hash, but you lose the salting which was only provided by the RSA step. There are easy solutions to this problem, of course: require a sufficiently large e, and include a salt in the hashed data. But if you have to rely on the rest of the design, why would you put RSA into the mix at all? This illustrates the risk of adding more stuff to a cryptographic algorithm: you can make it weaker even if you thought you were adding a security feature.
To summarize:
- Your proposal lacks a key property of password hashing functions: expensiveness.
- It may be possible to modify this proposal to have this property, but it is not clear how.
- You add a potential attack vector (algebraic relations); defenses exist but it is not clear what you are gaining.
