I'm trying to implement a self signed x509 certificate that uses a post-quantum (PQ) public key algorithm as the public key algorithm. I looked at the openssl library in c, and the way it's done using RSA. I'm essentially trying to replicate the same format. From what I've seen in the openssl library, RSA and a couple of other supported algorithms are integrated in the crypto EVP layer (the key is stored as EVP_PKEY). The functions in the openssl library that I'm trying to use are X509_REQ_set_pubkey(X509_REQ *x, EVP_PKEY *pkey), and some other functions that have very similar inputs. Is there a way that I can integrate the PQ algorithm into the EVP layer? If not, is there any way around using the EVP layer that would achieve the same goal?
I have tried looking into the evp source code in the openssl library. It seems it only supports certain algorithms such as RSA, EC... I'm not sure if it's possible to incorporate the PQ algorithm into the EVP layer. I'm following along the example in this link: (https://www.codepool.biz/how-to-use-openssl-to-generate-x-509-certificate-request.html) to generate the certificate. Instead of RSA key, I just plug in the PQ algorithm key. So far when I create my certificate, it's always outputted in the wrong format.
I'm using this command: openssl x509 -in x509Req.pem -text -noout to read the certificate generate. It always shows the error message "unable to load certificate \n 140688586052032:error:0906D06C:PEM routines:PEM_read_bio:no start line:../crypto/pem/pem_lib.c:691:Expecting: TRUSTED CERTIFICATE".