2

I'm using the RSA BSAFE Crypto-J 6.2 JCE provider in Java as SSL/TLS server (in FIPS 140 Compliant Mode for SunJSSE).

when I use any elliptic curve based key exchange mechanism the handshake fails when using a browser (Firefox and Chrome) as client.

In my specific case I use TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256. The "Client Hello" consists of three named curves:

Elliptic curve: secp256r1 (0x0017)
Elliptic curve: secp384r1 (0x0018)
Elliptic curve: secp521r1 (0x0019)

The resulting Server Hello always returns with an Unassigned (0xFFFF) Named Curve. As a result the browser (Firefox) terminates the connection with an error: "sec_error_unsupported_elliptic_curve"

Anyone who knows what I'm doing wrong in configuring server part?

-- This is a follow-up question to this question

Edit 1

Here is how I setup the HTTPS server (I used com.sun.net.httpserver.HttpsServer):

HttpsServer httpsServer = HttpsServer.create(address, 1);
SSLContext sslContext = SSLContext.getInstance("TLS");

// keystore
char[] password = "******".toCharArray();
KeyStore ks = KeyStore.getInstance("PKCS12");
FileInputStream fis = new FileInputStream("mykeystore.p12");
ks.load(fis, password);

// key manager factory
KeyManagerFactory kmf = KeyManagerFactory.getInstance("SunX509");
kmf.init(ks, password);

// trust manager factory
TrustManagerFactory tmf = TrustManagerFactory.getInstance("SunX509");
tmf.init(ks);

// HTTPS context
sslContext.init(kmf.getKeyManagers(), tmf.getTrustManagers(), new SecureRandom());

httpsServer.setHttpsConfigurator(new HttpsConfigurator(sslContext) {
    public void configure(HttpsParameters params) {
        try {
            SSLContext c = getSSLContext();
            SSLEngine engine = c.createSSLEngine();
            params.setNeedClientAuth(false);

            params.setCipherSuites(enabledCipherSuites);
            params.setProtocols(enabledProtocols);

            SSLParameters defaultSSLParameters = c.getDefaultSSLParameters();
            params.setSSLParameters(defaultSSLParameters);
        } catch (Exception ex) {
            ex.printStackTrace();
        }
    }
});

Here are the relevant changes in java.security:

...
com.rsa.cryptoj.fips140initialmode=FIPS140_SSL_MODE
...
security.provider.1=com.rsa.jsafe.provider.JsafeJCE
security.provider.2=sun.security.provider.Sun
security.provider.3=sun.security.rsa.SunRsaSign
security.provider.4=sun.security.ec.SunEC
security.provider.5=com.sun.net.ssl.internal.ssl.Provider JsafeJCE
security.provider.6=sun.security.jgss.SunProvider
security.provider.7=com.sun.security.sasl.Provider
security.provider.8=org.jcp.xml.dsig.internal.dom.XMLDSigRI
security.provider.9=sun.security.smartcardio.SunPCSC
...
keystore.type=pkcs12
...

In addition I removed SunJCE from the provider list.

Edit 2

This is the Client Hello

Secure Sockets Layer
    TLSv1 Record Layer: Handshake Protocol: Client Hello
        Content Type: Handshake (22)
        Version: TLS 1.0 (0x0301)
        Length: 129
        Handshake Protocol: Client Hello
            Handshake Type: Client Hello (1)
            Length: 125
            Version: TLS 1.0 (0x0301)
            Random
                GMT Unix Time: Aug 24, 2015 13:14:00.000000000 W. Europe Daylight Time
                Random Bytes: 33a17c871d7229ae33b9f62c99cfa79cb43e6b640efe18bc...
            Session ID Length: 0
            Cipher Suites Length: 58
            Cipher Suites (29 suites)
                Cipher Suite: TLS_EMPTY_RENEGOTIATION_INFO_SCSV (0x00ff)
                Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA (0xc00a)
                Cipher Suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (0xc014)
                Cipher Suite: TLS_DHE_RSA_WITH_AES_256_CBC_SHA (0x0039)
                Cipher Suite: TLS_DHE_DSS_WITH_AES_256_CBC_SHA (0x0038)
                Cipher Suite: TLS_ECDH_RSA_WITH_AES_256_CBC_SHA (0xc00f)
                Cipher Suite: TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA (0xc005)
                Cipher Suite: TLS_RSA_WITH_AES_256_CBC_SHA (0x0035)
                Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA (0xc009)
                Cipher Suite: TLS_ECDHE_ECDSA_WITH_RC4_128_SHA (0xc007)
                Cipher Suite: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (0xc013)
                Cipher Suite: TLS_ECDHE_RSA_WITH_RC4_128_SHA (0xc011)
                Cipher Suite: TLS_DHE_RSA_WITH_AES_128_CBC_SHA (0x0033)
                Cipher Suite: TLS_DHE_DSS_WITH_AES_128_CBC_SHA (0x0032)
                Cipher Suite: TLS_ECDH_RSA_WITH_AES_128_CBC_SHA (0xc00e)
                Cipher Suite: TLS_ECDH_RSA_WITH_RC4_128_SHA (0xc00c)
                Cipher Suite: TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA (0xc004)
                Cipher Suite: TLS_ECDH_ECDSA_WITH_RC4_128_SHA (0xc002)
                Cipher Suite: TLS_RSA_WITH_AES_128_CBC_SHA (0x002f)
                Cipher Suite: TLS_RSA_WITH_RC4_128_SHA (0x0005)
                Cipher Suite: TLS_RSA_WITH_RC4_128_MD5 (0x0004)
                Cipher Suite: TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA (0xc008)
                Cipher Suite: TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA (0xc012)
                Cipher Suite: TLS_DHE_RSA_WITH_3DES_EDE_CBC_SHA (0x0016)
                Cipher Suite: TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA (0x0013)
                Cipher Suite: TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA (0xc00d)
                Cipher Suite: TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA (0xc003)
                Cipher Suite: SSL_RSA_FIPS_WITH_3DES_EDE_CBC_SHA (0xfeff)
                Cipher Suite: TLS_RSA_WITH_3DES_EDE_CBC_SHA (0x000a)
            Compression Methods Length: 1
            Compression Methods (1 method)
                Compression Method: null (0)
            Extensions Length: 26
            Extension: elliptic_curves
                Type: elliptic_curves (0x000a)
                Length: 8
                Elliptic Curves Length: 6
                Elliptic curves (3 curves)
                    Elliptic curve: secp256r1 (0x0017)
                    Elliptic curve: secp384r1 (0x0018)
                    Elliptic curve: secp521r1 (0x0019)
            Extension: ec_point_formats
                Type: ec_point_formats (0x000b)
                Length: 2
                EC point formats Length: 1
                Elliptic curves point formats (1)
                    EC point format: uncompressed (0)
            Extension: SessionTicket TLS
                Type: SessionTicket TLS (0x0023)
                Length: 0
                Data (0 bytes)
            Extension: next_protocol_negotiation
                Type: next_protocol_negotiation (0x3374)
                Length: 0

... and here is the corresponding Server Hello

Secure Sockets Layer
    TLSv1 Record Layer: Handshake Protocol: Multiple Handshake Messages
        Content Type: Handshake (22)
        Version: TLS 1.0 (0x0301)
        Length: 1762
        Handshake Protocol: Server Hello
            Handshake Type: Server Hello (2)
            Length: 77
            Version: TLS 1.0 (0x0301)
            Random
                GMT Unix Time: Aug 24, 2015 13:12:33.000000000 W. Europe Daylight Time
                Random Bytes: 92d7f2c34202efa7d32934f37fe743795d66c23fe4d75720...
            Session ID Length: 32
            Session ID: 55dafc214d42a0c024fd6770ba008cfedb03b129a0c12cae...
            Cipher Suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (0xc014)
            Compression Method: null (0)
            Extensions Length: 5
            Extension: renegotiation_info
                Type: renegotiation_info (0xff01)
                Length: 1
                Renegotiation Info extension
                    Renegotiation info extension length: 0
        Handshake Protocol: Certificate
            Handshake Type: Certificate (11)
            Length: 1342
            Certificates Length: 1339
            Certificates (1339 bytes)
                Certificate Length: 811
                Certificate (id-at-countryName=DE,id-at-stateOrProvinceName=BW,id-at-localityName=*************,id-at-organizationName=******************,id-at-organizationalUnitName=***************,id-at-commonName=****************)
                Certificate Length: 522
                Certificate (id-at-commonName=swdevbbn)
        Handshake Protocol: Server Key Exchange
            Handshake Type: Server Key Exchange (12)
            Length: 327
            EC Diffie-Hellman Server Params
                Curve Type: named_curve (0x03)
                Named Curve: Unknown (0xffff)
                Pubkey Length: 65
                Pubkey: 04d7477d158ae38a62576e93b5a0b681494c4f6467ada53f...
                Signature Length: 256
                Signature: 4b415df019b2e0119778e3418d7a890753dd6552253367ed...
        Handshake Protocol: Server Hello Done
            Handshake Type: Server Hello Done (14)
  • Can you provide the relevant TLS configurations? – RoraΖ Aug 21 '15 at 17:01
  • Can you post the clienthello and serverhello? You can censor out the domain name from sni. – Z.T. Aug 24 '15 at 10:57
  • Pedantically, ServerKeyExchange (not ServerHello) has an ECDHE public key, probably for P-256, but it says 0xffff curve instead. That's a bug in BSAFE. You should contact EMC for support. Is it open source, can we look at the code? Also, does this version of BSAFE use Dual_EC_DRBG as its CSPRNG? If it does, you should change that. – Z.T. Aug 24 '15 at 21:20
  • Ok, thanks! I'll get in contact with EMC support. I was hoping that I just misconfigured something (BSAFE provider). From what I know, version 6.2 of BSAFE Crypto-J JCE provider does no longer use Dual_EC_DRBG as CSPRNG. – Cris Aug 24 '15 at 21:42
1

The issue described here has been fixed with RSA BSAFE Crypto-J 6.2.1 JCE provider.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.