What encoding is the modulus in RSA cryptography stored as in a .pem file? Is it something similar to base64 of the modulus?
I can retrieve the modulus using the following:
openssl rsa -in pubkey.pem -pubin -modulus
But what does it actually do? If I have the product of two prime numbers(n) in decimal form then how should I encode it so that the rsa utility can decode it?
There are two commonly used formats:
You can inspect them both for example with openssl asn1parse command or online decoder.
This question (and answer) is a subset of https://stackoverflow.com/questions/1193529/how-to-store-retreieve-rsa-public-private-key/13104466#13104466 . To compress and tailor that:
For an RSA public key you need the modulus and the exponent. It is very common to use either 3 or 65537 (aka F4, the fourth Fermat prime) as the exponent, but not mandatory or universal.
OpenSSL library can support either RSA-specific PKCS#1/rfc3447 key format, or the generic and more useful and common X.509 "SPKI" format, with either one stored in DER or PEM. The commandline utility rsa by default does SPKI PEM, but you can select the other formats with options, see the manpage (online at http://www.openssl.org/docs/apps/rsa.html ).
(As @zakjan says) DER is an encoding of ASN.1, and PEM is DER (further) encoded as base64 (the whole thing, not individual fields like the modulus), plus BEGIN and END lines, which indicate the format, in this case RSA PUBLIC KEY for PKCS#1 versus just PUBLIC KEY for SPKI. To encode PKCS#1 in DER, encode the modulus and exponent as ASN.1 INTEGERs, remembering that RSA moduli are almost always sized as a multiple of 8 (e.g. 1024, 2048) and so require a leading 00 octet, then encode in a SEQUENCE. For SPKI, further embed in SEQUENCE with an AlgorithmIdentifier identifying RSA, per rfc3279.
Alternatively, you can use OpenSSL library to encode for you. Put the two numbers in "bignum" (BN) objects, point to those from an RSA object (leaving the other fields initialized to empty), and either write that with the (unpreferred) RSAPublicKey routines, or better set it in an (empty) EVP_PKEY object with EVP_PKEY_set1_RSA and write with d2i_PublicKey{_fp,bio} for DER or PEM_write[_bio]_PublicKey for PEM.
