8

The server environment: Windows Server 2012 R2 + IIS8.5.

We are testing three self-signed certificates created by: C#(System.Security.Cryptography.X509Certificates), makecert+pvk2pfx and openssl respectively.

A strange thing happens:
When IIS binds certs created by makecert or openssl, the connection from client browser works well.
When IIS bind certs created by C#, the connection is reset by server immediately after it receives Client Hello from client. Traffic captured by Wireshark listed as below: 

1   0.000000    10.33.94.249    10.33.92.25     TCP 66  8740->7443 [SYN] Seq=0 Win=8192 Len=0 MSS=1460 WS=256 SACK_PERM=1  
2   0.015387    10.33.92.25     10.33.94.249    TCP 66  7443->8740 [SYN, ACK] Seq=0 Ack=1 Win=8192 Len=0 MSS=1460 WS=256 SACK_PERM=1  
3   0.015663    10.33.94.249    10.33.92.25     TCP 60  8740->7443 [ACK] Seq=1 Ack=1 Win=65536 Len=0  
4   0.015845    10.33.94.249    10.33.92.25     SSL 271 **Client Hello**  
5   0.017408    10.33.92.25     10.33.94.249    TCP 54  7443->8740 **[RST, ACK]** Seq=1 Ack=218 Win=0 Len=0  

SSL Record Layer: Handshake Protocol: Client Hello  
Content Type: Handshake (22)  
Version: TLS 1.0  
    Handshake Protocol: Client Hello  
    Handshake Type: Client Hello (1)  
    Version: TLS 1.2

I have confirmed the three certs created in different ways has the same extensions (Enhanced Key Usage, Key Usage, Subject Key Identifier) and all have private keys.

----Update 1----
Per suggestion, I uploaded two certificates with private key included.
The bad one created by C#: http://www.wikiupload.com/RQRVG8J4PHJUS8J
The good one created by openssl: http://www.wikiupload.com/1RE4E7ASMLEX1EC
Password for key is 1.

----Update 2----
IIS log only shows success connection with the good cert. Failed connection with the bad cert does not show in the log.

Any further idea to debug?

----Update 3----
Add the details of Client Hello

No.     Time           Source                Destination           Protocol Length Info
      4 0.015845000    10.33.94.249          10.33.92.25           SSL      271    Client Hello

Frame 4: 271 bytes on wire (2168 bits), 271 bytes captured (2168 bits) on interface 0
Ethernet II, Src: Dell_9c:c4:57 (34:17:eb:9c:c4:57), Dst: Vmware_62:c8:76 (00:0c:29:62:c8:76)
Internet Protocol Version 4, Src: 10.33.94.249 (10.33.94.249), Dst: 10.33.92.25 (10.33.92.25)
Transmission Control Protocol, Src Port: 8740 (8740), Dst Port: 7443 (7443), Seq: 1, Ack: 1, Len: 217
Secure Sockets Layer
    SSL Record Layer: Handshake Protocol: Client Hello
        Content Type: Handshake (22)
        Version: TLS 1.0 (0x0301)
        Length: 212
        Handshake Protocol: Client Hello
            Handshake Type: Client Hello (1)
            Length: 208
            Version: TLS 1.2 (0x0303)
            Random
            Session ID Length: 0
            Cipher Suites Length: 22
            Cipher Suites (11 suites)
                Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 (0xc02b)
                Cipher Suite: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (0xc02f)
                Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA (0xc00a)
                Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA (0xc009)
                Cipher Suite: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (0xc013)
                Cipher Suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (0xc014)
                Cipher Suite: TLS_DHE_RSA_WITH_AES_128_CBC_SHA (0x0033)
                Cipher Suite: TLS_DHE_RSA_WITH_AES_256_CBC_SHA (0x0039)
                Cipher Suite: TLS_RSA_WITH_AES_128_CBC_SHA (0x002f)
                Cipher Suite: TLS_RSA_WITH_AES_256_CBC_SHA (0x0035)
                Cipher Suite: TLS_RSA_WITH_3DES_EDE_CBC_SHA (0x000a)
            Compression Methods Length: 1
            Compression Methods (1 method)
                Compression Method: null (0)
            Extensions Length: 145
            Extension: server_name
                Type: server_name (0x0000)
                Length: 34
                Server Name Indication extension
            Extension: renegotiation_info
                Type: renegotiation_info (0xff01)
                Length: 1
                Renegotiation Info extension
            Extension: elliptic_curves
                Type: elliptic_curves (0x000a)
                Length: 8
                Elliptic Curves Length: 6
                Elliptic curves (3 curves)
            Extension: ec_point_formats
                Type: ec_point_formats (0x000b)
                Length: 2
                EC point formats Length: 1
                Elliptic curves point formats (1)
            Extension: SessionTicket TLS
                Type: SessionTicket TLS (0x0023)
                Length: 0
                Data (0 bytes)
            Extension: next_protocol_negotiation
                Type: next_protocol_negotiation (0x3374)
                Length: 0
            Extension: Application Layer Protocol Negotiation
                Type: Application Layer Protocol Negotiation (0x0010)
                Length: 41
                ALPN Extension Length: 39
                ALPN Protocol
            Extension: status_request
                Type: status_request (0x0005)
                Length: 5
                Certificate Status Type: OCSP (1)
                Responder ID list Length: 0
                Request Extensions Length: 0
            Extension: signature_algorithms
                Type: signature_algorithms (0x000d)
                Length: 18
                Signature Hash Algorithms Length: 16
                Signature Hash Algorithms (8 algorithms)
                    Signature Hash Algorithm: 0x0401
                        Signature Hash Algorithm Hash: SHA256 (4)
                        Signature Hash Algorithm Signature: RSA (1)
                    Signature Hash Algorithm: 0x0501
                        Signature Hash Algorithm Hash: SHA384 (5)
                        Signature Hash Algorithm Signature: RSA (1)
                    Signature Hash Algorithm: 0x0201
                        Signature Hash Algorithm Hash: SHA1 (2)
                        Signature Hash Algorithm Signature: RSA (1)
                    Signature Hash Algorithm: 0x0403
                        Signature Hash Algorithm Hash: SHA256 (4)
                        Signature Hash Algorithm Signature: ECDSA (3)
                    Signature Hash Algorithm: 0x0503
                        Signature Hash Algorithm Hash: SHA384 (5)
                        Signature Hash Algorithm Signature: ECDSA (3)
                    Signature Hash Algorithm: 0x0203
                        Signature Hash Algorithm Hash: SHA1 (2)
                        Signature Hash Algorithm Signature: ECDSA (3)
                    Signature Hash Algorithm: 0x0402
                        Signature Hash Algorithm Hash: SHA256 (4)
                        Signature Hash Algorithm Signature: DSA (2)
                    Signature Hash Algorithm: 0x0202
                        Signature Hash Algorithm Hash: SHA1 (2)
                        Signature Hash Algorithm Signature: DSA (2)

After reviewing this and the certificates I found a suspicious point: the signature algorithm.

C# cert:
Signature algorithm: sha512RSA
Signature hash algorithm: sha512
Thumbprint algorithm: sha1

openssl cert:
Signature algorithm: sha256RSA
Signature hash algorithm: sha256
Thumbprint algorithm: sha1

IIS cert:
Signature algorithm: sha1RSA
Signature hash algorithm: sha1
Thumbprint algorithm: sha1

Among them only C# cert uses sha512 and in Client Hello's cipher suites there is no sha512 specified. So maybe Server found client does not support SHA512 used in C# cert so reset the connection?

Improve this question
6
  • 1
    Obviously there is something in the certificate provided by C# which IIS does not like. If you need help you need to provide the certificate and key. The packet capture will no help since it only shows that there is some problem. You might also look into log messages/events generated by IIS which might show the cause of the problem.
    – Steffen Ullrich
    Aug 14, 2015 at 1:43
  • 1
    I'm a bit confused. The server sends a reset after the Client Hello message. The Client Hello message type does not include a certificate at all. If the server isn't responding with a Server Hello, then there is something wrong with the initial proposed communication parameters.
    – RoraΖ
    Aug 14, 2015 at 13:20
  • 1
    Or the server desides it cannot satisfy the client's requirements. For example client requires signature algo, hash algo, ecc curve or ecc point format the server cannot satisfy. The server should send a fatal alert and close connection.
    – Z.T.
    Aug 14, 2015 at 14:20
  • @SteffenUllrich certificates with keys are uploaded. Will figure out where is the related log in IIS and check if there's anything valuable.
    – Lee
    Aug 14, 2015 at 18:28
  • @raz You mean wrong parameters in initial communication from client? The strange thing is here, the client sends the same connection to server, the only difference is server binding with different certs.
    – Lee
    Aug 14, 2015 at 23:29

1 Answer 1

Reset to default
3

EDIT: You nailed it.

Among them only C# cert uses sha512 and in Client Hello's cipher suites there is no sha512 specified. So maybe Server found client does not support SHA512 used in C# cert so reset the connection?

Yup. Details below.

SHA-512 confusion?

One striking difference between the certs is the signature algorithms:

  • C-Sharp certificate: Signature Algorithm: sha512WithRSAEncryption
  • OpenSSL certificate: Signature Algorithm: sha256WithRSAEncryption

And there seems to be something buggy going on there with IIS and SHA-512. I don't really understand it, but there's buzz in the right places:

Suggestion: use SHA-256

Try to make C-Sharp generate a SHA-256 signed certificate. Check if this fixes it.

Improve this answer
2
  • The links you provide are good source, thanks. One more question: In Client Hello in my update 3, why SSL Record Layer's version is TLS 1.0 while Handshake Protocol's version is TLS 1.2? Which is the actual TLS version the client is asking to the server?
    – Lee
    Aug 17, 2015 at 5:16
  • 1.2. See this answer by Tom Leek: security.stackexchange.com/questions/29314/what-is-the-significance-of-the-version-field-in-a-tls-1-1-clienthello-message
    – StackzOfZtuff
    Aug 17, 2015 at 11:20

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .