I have installed openssl and now the rsa_test.c is running fine. What I want to do is:

  1. Open any exe or dll digital certificate. Extract the Thumbprint and PublicKey. The public key contains the exponent too (if you are familiar with ASN1 encoding).

  2. Now I have already calculated SHA-1 hash of the file and compared it to the MD field to find it correct. So now I want to pass the Thumbprint, PublicKey->m and PublicKey->e to RSA_public_decrypt function to calculate the decrypted SHA-1 hash of the file.

What Padding Algorithm should I pass to the function? When I pass RSA_NO_PADDING, it decrypts but give me the 128-bytes long hash which is not at all correct. For all other padding algorithm options it returns the error of padding not found.

Would you please teach me or tell me the correct parameters to pass through the function with an example or good link to example?

closed as off topic by Jeff Ferland Feb 1 '13 at 23:41

Questions on Information Security Stack Exchange are expected to relate to Information security within the scope defined by the community. Consider editing the question or leaving comments for improvement if you believe the question can be reworded to fit within the scope. Read more about reopening questions here. If this question can be reworded to fit the rules in the help center, please edit the question.