I have a hashed password and the plaintext password and i want to know the exact algorithm that's been used. I have the input and the output of the hash/encription algorithm. How would I know what algorithm is used?
There is only one solution: try all known algorithms on your input until you find your output.
I've never really tried to do this, but I've hung around with people who do that all the time. What I gather is that it is a bit of an art.
In some cases you can tell by length. For example SHA1 will produce 20 byte hashes (plus salt). Other schemes will have different outputs. Also you might find this hashes with separators in them (for things like salts and so on). Different systems for creating passwords have various conventions.
The people behind John the Ripper have a list of samples, but it isn't very complete. But you should at least be able to use it to eliminate some possibilities.
If you have a large dump of hashes, try testing them with some of the plausible formats using just the top 500 or so most common passwords. ()'123456', "Password1', etc). Once you break one of the hashes with something like that, you will know what scheme is used.
If you want to get some idea of how the experts do this (often with very tricky hashes) you should read some of the write-ups from the teams who have competed in Korelogic's "Crack Me If You Can" competitions.
You might try search or posting to the john-users mailing list. I have found them extremely helpful in these kinds of things. There is also an active community of haschat users as well.
But because this is an art, most people will need to know more about your hash (where it came from, what it looks like) before they can give you specific help.