SSL/TLS - Certificate chain validation result in "Invalid authentication type: DHE_RSA"

Question

I have an application that connects to a remote web service deployed on HTTPS. Our application uses FIPS-140 Level-1 for SSL/TLS connectivity, and is based on JCE providers of RSA (the company).

com.rsa.jsse.JsseProvider.JsseProvider()
RsaJsse version 6.14

com.rsa.jsafe.provider.JsafeJCE.JsafeJCE()
JsafeJCE version 6.11

Recently the web service team made changes on their end to enable DH ciphers. After that change, the cipher that gets selected for SSL is TLS_DHE_RSA_WITH_AES_256_GCM_SHA384. However, the certificate chain validation seems to fail with below exception:

java.lang.IllegalArgumentException: Invalid authentication type: DHE_RSA
    at com.rsa.sslj.x.cj.checkClientTrusted(Unknown Source)
    at com.example.MyX509TrustManager.checkServerTrusted(ReloadableX509TrustManager.java:65)
    at com.rsa.sslj.x.aE.a(Unknown Source)
    at com.rsa.sslj.x.bg.a(Unknown Source)
    at com.rsa.sslj.x.bg.a(Unknown Source)
    at com.rsa.sslj.x.bg.a(Unknown Source)
    at com.rsa.sslj.x.aH.a(Unknown Source)
    at com.rsa.sslj.x.aH.a(Unknown Source)
    at com.rsa.sslj.x.ap.c(Unknown Source)
    at com.rsa.sslj.x.ap.a(Unknown Source)
    at com.rsa.sslj.x.ap.j(Unknown Source)
    at com.rsa.sslj.x.ap.i(Unknown Source)
    at com.rsa.sslj.x.ap.h(Unknown Source)
    at com.rsa.sslj.x.aS.startHandshake(Unknown Source)
    at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:261)
    at org.apache.http.impl.conn.HttpClientConnectionOperator.connect(HttpClientConnectionOperator.java:118)

Can anyone please guide me what is the meaning of "Invalid authentication type: DHE_RSA"? Prior to DH Cipher enablement in back end webservice, this authentication type used to be "RSA", and we never faced any issue.

I read in the comments of this question that this error can come if algorithm is not FIPS-140 approved. Can that be the case here?

Please note that method com.example.MyX509TrustManager.checkServerTrusted does not do anything special - it is written to skip certificate chain validation in certain special cases, else it will delegate to java.net.ssl.TrustManager#checkClientTrusted - in this case TrustManager is instance of an RSA class whose name is obfuscated in RSA jars.

UPDATE:

It seems that com.rsa.sslj.x.cj.checkClientTrusted expects the algorithm used in public key should be DHE_RSA, since the algorithm used in remote web service's certificate chain is RSA - it reports an error saying Invalid authentication type. Remote web service team had recently added dhparam to their SSL configuration to avoid logjam vulnerability, but they have not done any changes to their certificate. Not sure why our SSL client has started failing to connect to remote service since then, and is there anything I need to do for our application or remote web services team need to something. Any inputs in this regard will be highly helpful

Answer 1

Your custom checkServerTrusted should not be forwarding to the standard checkClientTrusted but to the standard checkServerTrusted. The authType for Server is a keyexchange name, but for Client it is an algorithm name -- THESE ARE NOT IN GENERAL THE SAME and in the case you just encountered they are and should be different.

FIPS140 only specifies and restricts algorithms, not higher-level constructs like TLS keyexchanges, ciphersuites, and protocols. DH is approved (by 800-56A) only with some constraints that don't work for TLS, but it is allowed for a case that covers DHE_RSA by the FIPS1402 Implementation Guidance D.8 and is listed as implemented that way in almost your JSAFE version at CMVP #2057. OTOH SP800-52 does restrict TLS for most of the same systems as FIPS140, and r1 in 2014 does not allow DHE_RSA, but #2057 predates 2014. So I'm pretty sure there's no FIPS problem here.