0

Our BizTalk installation consists of two load balanced nodes B1 (10.137.10.1) and B2 (10.137.10.2) which process and route incoming SOAP messages.

The processing is nothing more than:

  • sending the SOAP request message to an external web service (WS) (out of our control),
  • wait for the reply
  • and route the response back to the original caller.

In front of the web service there is an nginx reverse proxy (RP, WS/RP: 145.21.186.179).

The connection between the BizTalk nodes (B1, B2) and the WS (or actually RP) will be HTTPS (TLSv1) with mutual/two-way authentication.

B1 works

All goes fine when messages are routed by node B1. A proper TLSv1 handshake is performed and eventually application data (the request and response message) are exchanged. See screenshot below.

Wire shark on node B1. (everything goes fine)

  • RP requests certificate at packet 63. (At 459 seconds mark.)
  • B1 answers at packet 69. (Also at 459 seconds mark.)

Wireshark on node B1

B2 falls silent

Messages processed by node B2 fail. We don't see a proper TLSv1 handshake. After the server requests the certificate from the client it goes silent.

Wire shark on node B2. (client goes silent after server certificate request)

  • RP requests certificate at packet 27. (At 96 second mark.)
  • B2 does not answer.
  • RP times out (?) and closes the connection at paket 31. (At 155 second mark.)

Wireshark on node B2

Question

Could you guys and girls think of one reason why this could happen? We checked the configuration on both B1 and B2 and we don't see deviations.

Also testing the connection with OpenSSL from node B2 works perfectly. Command was this:

openssl.exe s_client -connect RPHOSTNAMEHERE:443 -state -tls1 -debug -cert client_ssl.pem -key client_ssl.pem

Could this be a problem with SChannel? What would be a reason to not send the client certificate upon server request?

Also the event logs don't give us any leads.

Update #1 sharing spurious retransmision from B2 screenshot

enter image description here

  • I can see some spurious message alert sent from the client to the server in the second screenshot. Can you share that one? – Limit Jun 14 '16 at 14:19
  • I think this question might get better answers on Server Fault. – Ajedi32 Jun 14 '16 at 14:24
  • @Limit Ok Done! – Martijn B Jun 14 '16 at 14:33
  • A reason might be if the no client certificate is configured at B2 or if the configuration is wrong, requires password or whatever. Since B2 is the one causing problems I would suggest to look further into this server (logs, configuration...). Tests with OpenSSL will not help much because this is a different TLS stack with different configuration. – Steffen Ullrich Jun 14 '16 at 15:47
  • You wrote " testing the connection with OpenSSL from node B1 works perfectly." But what about from B2 to RP with OpenSSL? – StackzOfZtuff Jun 15 '16 at 12:39
1

From the Wireshark trace on B2, we see that the client is simply not responding at all; after one minute, the server grows impatient and closes the connection.

This is probably not a problem of certificate availability on B2: if SChannel simply believed that it had no appropriate certificate, then it would send back an empty Certificate handshake message. At that point, B2 perfectly knows that it is its turn to talk, so if it does not, then that means that it is occupied elsewhere.

One of the following may happen:

  • There are several matching certificates on B2, and the software on B2 believes itself to be "interactive"; it is thus displaying a certificate selection popup and waits for the user to choose the certificate. I have seen cases where software running as a service account would "show" a popup on a hidden desktop (not the one of the user currently logged in).

  • The private key is access-protected. It is possible to enable password-based access control on private keys in Windows CryptoAPI (basically, the private key is stored with password-based encryption). In that case, any access will trigger a popup requesting the password entry. For non-interactive systems, this won't go well... (see above). Note that the private key is needed for the CertificateVerify message from the client, after the Certificate and ClientKeyExchange messages, but since these three messages are all "handshake", SChannel will group them in a single record, so in that situation it would be normal not to see the Certificate and ClientKeyExchange messages.

  • The client is busy validating the server's certificate. Such a validation may entail talking to a lot of other systems, to gather additional intermediate CA and obtain CRL or OCSP responses. These are outgoing requests, so your network installation might block them. Try making a Wireshark trace that includes all traffic, not just the specific TCP/SSL connection; in particular, DNS requests that get no answer should be investigated.

A quick test would be to try to connect from B2 to the target server WS with Internet Explorer. IE will use SChannel (contrary to openssl.exe). Run IE with an account that has access to the expected client certificate and key. This will show you any "interactive" feature.

  • +1 great leads. Thank you very much. I am going to check this tomorrow with the responsible sysadmin. – Martijn B Jun 15 '16 at 13:31
  • @MartijnB: Also: wfetch.exe (from IIS60ResKit) might be worth a try. I think it can handle client certs. – StackzOfZtuff Jun 15 '16 at 13:42
  • Found It!!!! The certificate was installed under the CurrentUser cert store with Strong Key protection enabled. This feature will prompt the user (service account) for consent which it never gets. On the B1 server this feature was disabled. Thank you so much! – Martijn B Jun 21 '16 at 10:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.