The server environment: Windows Server 2012 R2 + IIS8.5.
We are testing three self-signed certificates created by: C#(System.Security.Cryptography.X509Certificates), makecert+pvk2pfx and openssl respectively.
A strange thing happens:
When IIS binds certs created by makecert or openssl, the connection from client browser works well.
When IIS bind certs created by C#, the connection is reset by server immediately after it receives Client Hello from client. Traffic captured by Wireshark listed as below:
1 0.000000 10.33.94.249 10.33.92.25 TCP 66 8740->7443 [SYN] Seq=0 Win=8192 Len=0 MSS=1460 WS=256 SACK_PERM=1
2 0.015387 10.33.92.25 10.33.94.249 TCP 66 7443->8740 [SYN, ACK] Seq=0 Ack=1 Win=8192 Len=0 MSS=1460 WS=256 SACK_PERM=1
3 0.015663 10.33.94.249 10.33.92.25 TCP 60 8740->7443 [ACK] Seq=1 Ack=1 Win=65536 Len=0
4 0.015845 10.33.94.249 10.33.92.25 SSL 271 **Client Hello**
5 0.017408 10.33.92.25 10.33.94.249 TCP 54 7443->8740 **[RST, ACK]** Seq=1 Ack=218 Win=0 Len=0
SSL Record Layer: Handshake Protocol: Client Hello
Content Type: Handshake (22)
Version: TLS 1.0
Handshake Protocol: Client Hello
Handshake Type: Client Hello (1)
Version: TLS 1.2
I have confirmed the three certs created in different ways has the same extensions (Enhanced Key Usage, Key Usage, Subject Key Identifier) and all have private keys.
----Update 1----
Per suggestion, I uploaded two certificates with private key included.
The bad one created by C#: http://www.wikiupload.com/RQRVG8J4PHJUS8J
The good one created by openssl: http://www.wikiupload.com/1RE4E7ASMLEX1EC
Password for key is 1.
----Update 2----
IIS log only shows success connection with the good cert. Failed connection with the bad cert does not show in the log.
Any further idea to debug?
----Update 3----
Add the details of Client Hello
No. Time Source Destination Protocol Length Info
4 0.015845000 10.33.94.249 10.33.92.25 SSL 271 Client Hello
Frame 4: 271 bytes on wire (2168 bits), 271 bytes captured (2168 bits) on interface 0
Ethernet II, Src: Dell_9c:c4:57 (34:17:eb:9c:c4:57), Dst: Vmware_62:c8:76 (00:0c:29:62:c8:76)
Internet Protocol Version 4, Src: 10.33.94.249 (10.33.94.249), Dst: 10.33.92.25 (10.33.92.25)
Transmission Control Protocol, Src Port: 8740 (8740), Dst Port: 7443 (7443), Seq: 1, Ack: 1, Len: 217
Secure Sockets Layer
SSL Record Layer: Handshake Protocol: Client Hello
Content Type: Handshake (22)
Version: TLS 1.0 (0x0301)
Length: 212
Handshake Protocol: Client Hello
Handshake Type: Client Hello (1)
Length: 208
Version: TLS 1.2 (0x0303)
Random
Session ID Length: 0
Cipher Suites Length: 22
Cipher Suites (11 suites)
Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 (0xc02b)
Cipher Suite: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (0xc02f)
Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA (0xc00a)
Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA (0xc009)
Cipher Suite: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (0xc013)
Cipher Suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (0xc014)
Cipher Suite: TLS_DHE_RSA_WITH_AES_128_CBC_SHA (0x0033)
Cipher Suite: TLS_DHE_RSA_WITH_AES_256_CBC_SHA (0x0039)
Cipher Suite: TLS_RSA_WITH_AES_128_CBC_SHA (0x002f)
Cipher Suite: TLS_RSA_WITH_AES_256_CBC_SHA (0x0035)
Cipher Suite: TLS_RSA_WITH_3DES_EDE_CBC_SHA (0x000a)
Compression Methods Length: 1
Compression Methods (1 method)
Compression Method: null (0)
Extensions Length: 145
Extension: server_name
Type: server_name (0x0000)
Length: 34
Server Name Indication extension
Extension: renegotiation_info
Type: renegotiation_info (0xff01)
Length: 1
Renegotiation Info extension
Extension: elliptic_curves
Type: elliptic_curves (0x000a)
Length: 8
Elliptic Curves Length: 6
Elliptic curves (3 curves)
Extension: ec_point_formats
Type: ec_point_formats (0x000b)
Length: 2
EC point formats Length: 1
Elliptic curves point formats (1)
Extension: SessionTicket TLS
Type: SessionTicket TLS (0x0023)
Length: 0
Data (0 bytes)
Extension: next_protocol_negotiation
Type: next_protocol_negotiation (0x3374)
Length: 0
Extension: Application Layer Protocol Negotiation
Type: Application Layer Protocol Negotiation (0x0010)
Length: 41
ALPN Extension Length: 39
ALPN Protocol
Extension: status_request
Type: status_request (0x0005)
Length: 5
Certificate Status Type: OCSP (1)
Responder ID list Length: 0
Request Extensions Length: 0
Extension: signature_algorithms
Type: signature_algorithms (0x000d)
Length: 18
Signature Hash Algorithms Length: 16
Signature Hash Algorithms (8 algorithms)
Signature Hash Algorithm: 0x0401
Signature Hash Algorithm Hash: SHA256 (4)
Signature Hash Algorithm Signature: RSA (1)
Signature Hash Algorithm: 0x0501
Signature Hash Algorithm Hash: SHA384 (5)
Signature Hash Algorithm Signature: RSA (1)
Signature Hash Algorithm: 0x0201
Signature Hash Algorithm Hash: SHA1 (2)
Signature Hash Algorithm Signature: RSA (1)
Signature Hash Algorithm: 0x0403
Signature Hash Algorithm Hash: SHA256 (4)
Signature Hash Algorithm Signature: ECDSA (3)
Signature Hash Algorithm: 0x0503
Signature Hash Algorithm Hash: SHA384 (5)
Signature Hash Algorithm Signature: ECDSA (3)
Signature Hash Algorithm: 0x0203
Signature Hash Algorithm Hash: SHA1 (2)
Signature Hash Algorithm Signature: ECDSA (3)
Signature Hash Algorithm: 0x0402
Signature Hash Algorithm Hash: SHA256 (4)
Signature Hash Algorithm Signature: DSA (2)
Signature Hash Algorithm: 0x0202
Signature Hash Algorithm Hash: SHA1 (2)
Signature Hash Algorithm Signature: DSA (2)
After reviewing this and the certificates I found a suspicious point: the signature algorithm.
C# cert:
Signature algorithm: sha512RSA
Signature hash algorithm: sha512
Thumbprint algorithm: sha1
openssl cert:
Signature algorithm: sha256RSA
Signature hash algorithm: sha256
Thumbprint algorithm: sha1
IIS cert:
Signature algorithm: sha1RSA
Signature hash algorithm: sha1
Thumbprint algorithm: sha1
Among them only C# cert uses sha512 and in Client Hello's cipher suites there is no sha512 specified. So maybe Server found client does not support SHA512 used in C# cert so reset the connection?
Client Hello
message. The Client Hello message type does not include a certificate at all. If the server isn't responding with a Server Hello, then there is something wrong with the initial proposed communication parameters.