How is a Digital signature of a X.509 generated?

Question

I was asked a question recently at a job interview about HTTPS and admitted my knowledge was fuzzy in the area, but gave a crack at it anyway, the question was how does someone know that when they connect to a server that the person or service say who they say they are? how can you trust them? I answered you will be given a certificate from the CA and this contains information that the connection is legitimate, this was apparently not satisfactory as they then pushed for more asking about how do you know they are who they say they are from this certificate? In my head i thought being provided a certificate from the a CA was enough.

The answer they were looking for was the Digital signature. Which i completely forgot about. Nevertheless, feeling like a complete dolt afterwards i decided to shore up some holes in my knowledge and did me some learnin'.

So to recap (please feel free to tell me if im missing anything)the Digital signature is created by hashing some data on the certificate then encrypting it using the CA private key, which the person interested in authenticating the validity of said certificate will generate a hash on the same data on the certificate used in the signature generation, then use the certificates public key to decrypt the digital signature obtaining the hash, and then comparing them.

Now where Im a little perplexed about is how does let say a client know which part of the certificate to hash to then compare it to the hash retrieved from using the certificates public key on the digital signature. Does x.509 have a standard or do they just hash everything in the certificate? or do they communicate somehow on how they generate the hash?

Answer 1

... which part of the certificate to hash ... Does x.509 have a standard or do they just hash everything in the certificate?

It's all defined in RFC 5280. In section 4.1.1.3. you will find:

4.1.1.3. signatureValue
The signatureValue field contains a digital signature computed upon the ASN.1 DER encoded tbsCertificate.

And to find out what this tbsCertificate is continue reading with section 4.1.2:

4.1.2. TBSCertificate
The sequence TBSCertificate contains information associated with the subject of the certificate and the CA that issued it. Every TBSCertificate contains the names of the subject and issuer, a public key associated with the subject, a validity period, a version number, and a serial number; some MAY contain optional unique identifier fields.....

The certificate itself is then the tbsCertificate, the signatureAlgorithm and the signature. Or in a more formally defined way at page 116:

Certificate  ::=  SEQUENCE  {
     tbsCertificate       TBSCertificate,
     signatureAlgorithm   AlgorithmIdentifier,
     signature            BIT STRING  }

TBSCertificate  ::=  SEQUENCE  {
     version         [0]  Version DEFAULT v1,
     serialNumber         CertificateSerialNumber,
     signature            AlgorithmIdentifier,
     issuer               Name,
     validity             Validity,
     subject              Name,
     ...