Using RSA_verify to verify a signature, why the first parameter?

Question

The RSA_verify function takes the following parameters:

int RSA_verify(int type, unsigned char*m, unsigned int m_len,
               unsigned char *sigbuf, unsigned int siglen, RSA* rsa);

the first parameter is the type of message digest algorithm, like NID_sha256.

Given that this code does not hash the message, the variable m is the hashed message, what purpose is there to passing in the first parameter?

Is the hash algorithm type stored somewhere, and is verify checking that the same algorithm is being used to verify as was used to sign? If so, this int has nothing to do with the function that was used, it could easily be wrong.

Answer 1

From the documentation you can see that the type is important in RSA_sign because different kinds of output are used depending on the type:

type denotes the message digest algorithm that was used to generate m. It usually is one of NID_sha1, NID_ripemd160 and NID_md5; see objects for details. If type is NID_md5_sha1, an SSL signature (MD5 and SHA1 message digests with PKCS #1 padding and no algorithm identifier) is created.

Similar the type needs to be known when verifying the signature because the implementation needs to know which format the signature is in order to parse it correctly. If you look at the source code you see special handling for NID_md5_sha1 and NID_mdc2.

Answer 2

The EMSA encoding mandated by PKCS#1 for calculating the message hash includes the OID of the hash functions in the message bytes. This is done to make it harder to use a different hashing scheme. This is described in RFC 3447 9.2.

The mixed MD5/SHA1 on the other hand is the standard format used with older TLS handshakes. It is not uncommon for crypto primitives to have some format or padding related parameters, especially since those are designed to protect against failures of naive implementations.