I have a SMTP server where all the users (for an unrelated reason) will have public keys tied to their username and registered with the server. I want to add a non-standard extension that uses these existing keys for authentication. My understanding of the SSH key based auth is that the server encrypts a random number and sends it to the client, the client then decrypts the number and sends it back to the server. The user is considered to have succeeded only if the original number and the number returned by the client match. I think there's also negotiation of a session key in there somewhere but that won't be necessary for my purposes.
Assuming that the asymmetric key algorithm (elGamal in this case) and the random number generator are both secure, are there additional intricacies of the SSH auth model that I need to take into account?
Additionally does this violate the "don't roll your own" principle? All the components aren't home made and their composition is at least an imitation of a well trusted standard, but it does seem to be going further that sticking a library in and calling it a day.