Why does SSL connection does not fail after modifying the certificate in editor like VIM or Gedit?

Question

I want to fail the SSL connection between a server, hosted using openssl s_server command with a certificate issued by self signed CA, and a client(Browser). To do so, there are various ways:

1. Change the date of system which out of effective date of system; this way I am able to get an error in browser saying certificate is not valid for this date.

2. I tried to modify the server certificate in editor like VIM, to be specific I modified the CN( common name) field, because as per SSL standard CN must be verified to make sure that client is talking to actual server it wants to talk to, according to RFC 2818. Strangely, I observe that after restarting the server with this modified certificate client(Mozila Browser) is still able to establish a SSL connection with server. I don't get it how?

Ideally, as per my under standing if one single bit of certificate is modified certificate verification should fail, as client will calculate the hash of contents of certificate and compare it with decrypted signed hash from public key of CA, as certificate CN is modified hash of the contents of certificate will be different for sure.

Am I missing some important concept or I doing anything wrong in order to fail that connection?

In additon to this if I create a new certificate with different CN from same CA and use it inplace of original certificate then SSL conection fails saying Unable to communicate securely with peer: requested domain name does not match the server's certificate

Test Setup Information:

1. Created a self signed root CA, with private a key ca.key, and a certificate ca.crt
2. Created a private key named server.key; generated a csr for that private key CN as testserver.com; using self signed CA, signed the csr to generate a server.crt
3. Appended server.key and server.crt to a single file server.pem
4. Started a https server openssl s_server -cert server.pem -www
5. Added 127.0.0.1 testserver.com in /etc/hosts
6. Hit url https:testserver.com:4433` from firefox browser. SSL handshake is complete, and able to see cipher suite and session information
7. Create a copy of server.pem modified-server.pem, and edited the modified-server.pem's CN field to `invalidtestserver.com in vi editor and saved it.
8. Restart the server with modified-server.pem, with command openssl s_server -cert modified-server.pem -www and SSL handshake is still successful and able to see same output as previous one in browser.

Answer 1

Meta: not a definite answer, but needs (extensive) formatting.

Did you edit a file in the format OpenSSL often uses that looks like this

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            d7:a9:8b:73:69:4f:fa:75
        Signature Algorithm: sha1WithRSAEncryption
        Issuer: C=US, ST=HI, O=Test Company, CN=test.example.com
        Validity
            Not Before: Sep 10 08:25:49 2016 GMT
            Not After : Oct 10 08:25:49 2016 GMT
        Subject: C=US, ST=HI, O=Test Company, CN=test.example.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:b9:1c:c3:c5:ad:ed:cb:86:91:75:09:8c:3a:ce:
                    59:89:1d:35:68:52:63:8f:d5:63:25:f7:30:df:60:
                    8f:6c:b7:dd:28:d7:6b:05:1e:36:dc:b5:92:ef:8d:
                    a4:80:2c:59:2f:0d:d7:0b:d9:b8:cb:9b:d3:35:ff:
                    46:ae:3e:d9:fb:b2:a6:15:59:48:92:d7:0d:58:ae:
                    4b:70:04:72:b4:52:62:16:ab:8c:fb:5b:1f:ca:bc:
                    70:de:65:ba:cd:28:46:a3:aa:61:d1:6d:42:6a:15:
                    b9:5f:73:42:34:b0:7e:cd:ca:b6:1e:50:e4:4d:d2:
                    ae:60:ec:08:7c:c8:09:64:7e:d1:5c:35:7b:3a:db:
                    20:cd:27:1d:d0:0a:5b:c3:68:f7:20:10:0b:87:7e:
                    33:17:1d:e9:da:57:13:fd:e5:ab:a7:0e:dc:96:2b:
                    a0:72:a9:c9:16:31:b2:4e:10:c5:e8:06:28:89:d1:
                    5e:15:1e:b6:fb:0a:b3:1f:85:0d:d2:44:ff:da:fb:
                    36:bd:57:fa:b4:df:42:73:94:64:b0:19:44:68:38:
                    aa:cd:3d:54:48:8e:ba:18:58:61:bd:37:76:51:6b:
                    60:0c:93:9f:9a:c1:37:42:a4:89:63:21:cf:b3:7e:
                    ee:1f:83:e1:cf:8a:b2:21:05:08:22:54:4d:42:94:
                    15:df
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Subject Key Identifier:
                74:81:FE:9F:5A:85:31:E8:FE:07:09:54:78:27:3B:1B:8B:3A:43:AA
            X509v3 Authority Key Identifier:
                keyid:74:81:FE:9F:5A:85:31:E8:FE:07:09:54:78:27:3B:1B:8B:3A:43:AA

            X509v3 Basic Constraints:
                CA:TRUE
    Signature Algorithm: sha1WithRSAEncryption
        5e:87:9b:67:4c:87:01:24:ff:f2:7d:b1:d2:ab:09:4d:08:d8:
        ca:94:db:a1:0d:df:74:50:61:89:6e:3d:14:d4:b6:9d:86:1e:
        2c:8a:b0:42:fa:5f:bc:4d:ed:f8:7b:ec:24:f9:d3:4c:af:6e:
        04:6c:55:0a:86:17:71:87:a0:a1:65:5d:1e:03:f9:2e:74:68:
        ad:23:cb:a6:36:58:db:02:f1:21:64:a5:5d:62:70:83:e2:70:
        3c:ac:19:3d:62:df:6f:fc:e4:05:03:b6:7e:7c:84:2c:27:b0:
        1d:32:6b:93:15:fe:88:37:16:d1:fe:5e:c1:b9:b3:d2:ab:e4:
        ec:f7:1c:9c:42:d7:6e:3f:37:da:58:4b:b0:ea:e3:9d:7f:0e:
        6c:3a:8c:cf:3b:fb:16:c1:b5:7f:fb:1e:c1:e9:3e:40:cb:47:
        1a:84:f4:45:5c:68:16:c8:77:8f:14:b2:d7:57:8a:9d:31:d3:
        c4:ff:60:d4:cf:bb:d4:21:8d:40:e9:12:c9:43:a5:67:78:75:
        9e:ca:b5:0a:2a:00:d8:18:9d:c7:47:69:ce:f0:5f:c2:7a:b1:
        3e:bc:73:48:0a:dc:a1:d2:29:6a:b7:03:6f:8d:47:e6:e0:79:
        93:a8:fc:03:dd:a6:cd:37:09:09:6a:65:b2:4d:c6:94:56:fb:
        18:cb:be:f8
-----BEGIN CERTIFICATE-----
MIIDazCCAlOgAwIBAgIJANepi3NpT/p1MA0GCSqGSIb3DQEBBQUAMEwxCzAJBgNV
BAYTAlVTMQswCQYDVQQIDAJISTEVMBMGA1UECgwMVGVzdCBDb21wYW55MRkwFwYD
VQQDDBB0ZXN0LmV4YW1wbGUuY29tMB4XDTE2MDkxMDA4MjU0OVoXDTE2MTAxMDA4
MjU0OVowTDELMAkGA1UEBhMCVVMxCzAJBgNVBAgMAkhJMRUwEwYDVQQKDAxUZXN0
IENvbXBhbnkxGTAXBgNVBAMMEHRlc3QuZXhhbXBsZS5jb20wggEiMA0GCSqGSIb3
DQEBAQUAA4IBDwAwggEKAoIBAQC5HMPFre3LhpF1CYw6zlmJHTVoUmOP1WMl9zDf
YI9st90o12sFHjbctZLvjaSALFkvDdcL2bjLm9M1/0auPtn7sqYVWUiS1w1Yrktw
BHK0UmIWq4z7Wx/KvHDeZbrNKEajqmHRbUJqFblfc0I0sH7NyrYeUORN0q5g7Ah8
yAlkftFcNXs62yDNJx3QClvDaPcgEAuHfjMXHenaVxP95aunDtyWK6ByqckWMbJO
EMXoBiiJ0V4VHrb7CrMfhQ3SRP/a+za9V/q030JzlGSwGURoOKrNPVRIjroYWGG9
N3ZRa2AMk5+awTdCpIljIc+zfu4fg+HPirIhBQgiVE1ClBXfAgMBAAGjUDBOMB0G
A1UdDgQWBBR0gf6fWoUx6P4HCVR4JzsbizpDqjAfBgNVHSMEGDAWgBR0gf6fWoUx
6P4HCVR4JzsbizpDqjAMBgNVHRMEBTADAQH/MA0GCSqGSIb3DQEBBQUAA4IBAQBe
h5tnTIcBJP/yfbHSqwlNCNjKlNuhDd90UGGJbj0U1Ladhh4sirBC+l+8Te34e+wk
+dNMr24EbFUKhhdxh6ChZV0eA/kudGitI8umNljbAvEhZKVdYnCD4nA8rBk9Yt9v
/OQFA7Z+fIQsJ7AdMmuTFf6INxbR/l7BubPSq+Ts9xycQtduPzfaWEuw6uOdfw5s
OozPO/sWwbV/+x7B6T5Ay0cahPRFXGgWyHePFLLXV4qdMdPE/2DUz7vUIY1A6RLJ
Q6VneHWeyrUKKgDYGJ3HR2nO8F/CerE+vHNICtyh0ilqtwNvjUfm4HmTqPwD3abN
NwkJamWyTcaUVvsYy774
-----END CERTIFICATE-----

and change the fields labelled Issuer and/or Subject?

If so, that had no effect at all. The actual PEM-format certificate is the lines from ----BEGIN to -----END only; everything else is comment data intended to be possibly helpful to a human reader, but ignored by programs.

To change the certificate actually used in a program, you need to change the base64 data inside the PEM block. This is not easy, and you may find it easier to just generate different certs from scratch.

• convert the PEM/base64 to binary (which openssl a little imprecisely calls DER), parse the binary as ASN.1 DER (which openssl just calls ASN1) to find the position(s) of the byte(s) you want to change, edit the binary to change those bytes, and convert the result back to PEM (optionally re-adding the comment info). OpenSSL can do the first second and last easily, and vim can apparently do hex editting with a little effort (I don't do it myself) or there are alternative tools.

  If you change a value to be longer or shorter, you need to change several length fields at other places in the cert (see the ASN.1 parse), and if you don't get them all correct the result will be unusable.

• parse the PEM (directly) as ASN.1 to find the position(s) you want to change; this is easy with openssl. Convert those positions to positions in the base64 4-character chunks (requires a little arithmetic) and locate those base64 chunks, convert each to binary manually, calculate the change, and convert back to base64 manually and substitute back into the base64 blob.

  This method cannot handle change in length of a value at all.

Moreover, as Steffen commented, if you change anything in the body of a cert but leave the signature, then (with overwhelming probability) the signature won't verify and the browser or other relier may not even look at the other fields. Conversely if you change only the signature, at least for RSA and DSA, the result usually won't be valid signature for any cert body at all. If you want to specifically test the relier for changing (a) field(s) in the body, you also need to generate a new, valid signature. That may be possible, but you'll need to be specific about your CA hierarchy, unless you mean you are using a self-signed entity cert and not a CA at all, and in either case I'll need some more research I'd rather not do on spec.