I'm using the RSA BSAFE Crypto-J 6.2 JCE provider in Java as SSL/TLS server (in FIPS 140 Compliant Mode for SunJSSE).

It seems like at least Chrome and Firefox cannot establish a connection if TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 (0xc02b) is used.

The error in Firefox when connecting to the server is: "sec_error_unsupported_elliptic_curve"

With the error message as hint I looked into the handshake (using Wireshark). In the "Client Hello" I see 3 curves:

Elliptic curve: secp256r1 (0x0017)
Elliptic curve: secp384r1 (0x0018)
Elliptic curve: secp521r1 (0x0019)

In the corresponding "Server Hello" message I see the named curve value 0xffff

Named Curve: Unknown (0xffff)

As a result the browser terminates the connection with the above mentioned error message.

If I use standard Java on the server side (no BSAFE Crypto-J provider) the named curve in the "Server Hello" is:

Named Curve: secp256r1 (0x0017)

... and the handshake finishes successfully.

  1. The question now is, what is the meaning if 0xffff? Does it mean that the server does not understand the set of curves suggested by the client? I looked into section 5.1.1 of RFC4492 but I couldn't find anything about the value of 0xffff
  2. If someone has experience with RSA BSAFE Crypto-J and Java running in FIPS mode. Anybody else who ran into that problem and found a solution.

The question now is, what is the meaning if 0xffff? Does it mean that the server does not understand the set of curves suggested by the client?

I don't think it officially means this. But I have a feeling that the implementation is still doing it with that intention.

Unassigned at IANA

The identifier 0xFFFF (65535 decimal) is marked as Unassigned in the official IANA registry.

Namely this also means that it is NOT Reserved or marked as special in any other way.

Re. enum definition

The TLS 1.2 RFC contains a definition of the syntax for enums:

RFC 5246 - The Transport Layer Security (TLS) Protocol, Version 1.2, section 4.5 Enumerateds:

One may optionally specify a value without its associated tag to
force the width definition without defining a superfluous element.

In the following example, Taste will consume two bytes in the data
stream but can only assume the values 1, 2, or 4.

   enum { sweet(1), sour(2), bitter(4), (32000) } Taste;

I'm reading RFC4492 Section 5.1.1 to mean the same thing.

| improve this answer | |
  • So 0xFFFF (65535 decimal) means unassigned. Thanks for the offical link to IANA registry! In addition, the definition of the syntax for enums was the missing link for me to understand it. – Cris Aug 20 '15 at 7:19

Look at RFC4492 Section 5.1.1 that you mention in your post, we see the following structure:

enum {
        sect163k1 (1), sect163r1 (2), sect163r2 (3),
        sect193r1 (4), sect193r2 (5), sect233k1 (6),
        sect233r1 (7), sect239k1 (8), sect283k1 (9),
        sect283r1 (10), sect409k1 (11), sect409r1 (12),
        sect571k1 (13), sect571r1 (14), secp160k1 (15),
        secp160r1 (16), secp160r2 (17), secp192k1 (18),
        secp192r1 (19), secp224k1 (20), secp224r1 (21),
        secp256k1 (22), secp256r1 (23), secp384r1 (24),
        secp521r1 (25),
        reserved (0xFE00..0xFEFF),
    } NamedCurve;

It would seem to me that 0xFFFF is a reserved value that is (maybe mistakenly?) sent to indicate that the server doesn't support ECC, or to indicate the end of the array of curves.

It's possible that the BSAFE Crypto-J provider isn't properly configured.

| improve this answer | |

I had the exact same issue, everything was working with Java 7 and when I upgraded the JRE to Java 8, I started getting this error sec_error_unsupported_elliptic_curve.

I fixed the issue by setting the JVM parameter com.sun.net.ssl.enableECC to false


| improve this answer | |
  • This would appear to be how to disable elliptic curve support in Java completely, which will get rid of the error message, but doesn't really answer the question asked. – Matthew Dec 7 '15 at 9:30
  • Right! Although I'm currently using the same work around, I don't like to completely disable ECC. Please see also my related post. RSA BSAFE provider returns unassigned Named Curve – Cris Dec 7 '15 at 11:23
  • A solution was requested to solve this problem and neither posts had a solution. This workaround solves the problem, you say you have been already using it, and I thought it might help others like me who stumbled on this. It works as a quick fix, I am also looking for a more "proper" fix. – Mircea Stan Dec 7 '15 at 14:48
  • You're right Mircea, I missed to share my findings regarding the workaround. – Cris Jan 21 '16 at 8:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.