Understanding Java Security Providers

Question

I'm hoping to use the below issue we came across recently to better understand Java's security providers, ciphers and encryption in general.

We got the following error during an SSL Handshake (using the InstallCert tool) from a Solaris 10 host, running Java 1.6.0_81 to one of two AD hosts:

]$ java InstallCert host1.example.com:636 
Loading KeyStore /usr/jdk/instances/jdk1.6.0/jre/lib/security/cacerts...
Opening connection to host1.example.com:636...
Starting SSL handshake...

javax.net.ssl.SSLException: java.lang.RuntimeException: Could not parse key values
        at com.sun.net.ssl.internal.ssl.Alerts.getSSLException(Alerts.java:190)
        at com.sun.net.ssl.internal.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1747)
        at com.sun.net.ssl.internal.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1708)
        at com.sun.net.ssl.internal.ssl.SSLSocketImpl.handleException(SSLSocketImpl.java:1691)
        at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1222)
        at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1199)
        at InstallCert.main(InstallCert.java:98)
Caused by: java.lang.RuntimeException: Could not parse key values
        at sun.security.pkcs11.P11Key$P11ECPublicKey.fetchValues(P11Key.java:954)
        at sun.security.pkcs11.P11Key$P11ECPublicKey.getW(P11Key.java:975)
        at com.sun.net.ssl.internal.ssl.ECDHClientKeyExchange.<init>(ECDHClientKeyExchange.java:40)
        at com.sun.net.ssl.internal.ssl.ClientHandshaker.serverHelloDone(ClientHandshaker.java:782)
        at com.sun.net.ssl.internal.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:241)
        at com.sun.net.ssl.internal.ssl.Handshaker.processLoop(Handshaker.java:593)
        at com.sun.net.ssl.internal.ssl.Handshaker.process_record(Handshaker.java:529)
        at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:943)
        at com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1188)
        at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1215)
        ... 2 more
Caused by: java.io.IOException: extra data given to DerValue constructor
        at sun.security.util.DerValue.init(DerValue.java:368)
        at sun.security.util.DerValue.<init>(DerValue.java:277)
        at sun.security.pkcs11.P11Key$P11ECPublicKey.fetchValues(P11Key.java:945)
        ... 11 more

The second host does not give any errors. The only difference I've been able to glean here is the cipher on host1.example.com -- it is AES256-SHA; cipher on host2.example.com is AES128-SHA.

]$ echo -n | openssl s_client -connect host1.example.com:636 | grep 'Cipher is'
New, TLSv1/SSLv3, Cipher is AES256-SHA

]$ echo -n | openssl s_client -connect host2.example.com:636 | grep 'Cipher is'
New, TLSv1/SSLv3, Cipher is AES128-SHA

My questions are:

• What is the purpose of Java security providers and what is their role in encryption?
• What is it about the cipher of AES256-SHA on host1.example.com that yields this error?

• We eventually commented out this security provider in java.security file to resolve the issue:

  security.provider.1=sun.security.pkcs11.SunPKCS11 ${java.home}/lib/security/sunpkcs11-solaris.cfg

So it appears that this combination of a buggy provider and Cipher was causing this issue? That link references "EC", and I'm not sure what that is.

Any pointers and insights much appreciated!

Answer 1

And more specifically ECDH is the Elliptic Curve variant of Diffie-Hellman, used here for key exchange. Almost certainly in the form of ECDHE, ephemeral ECDH, but we can't confirm that from the information here.

You can't say "the" cipher on the server; practically all SSL/TLS implementations support multiple ciphersuites (abbreviated, imprecisely, ciphers). Normally the client offers a big list of suites that the client supports, and the server chooses the "best" one among them that the server also supports, where exactly what is "best" is complicated and there are (many) other questions on that. Even if I ass-u-me AD means Windows Active Directory, I don't know what suites that supports and I believe it depends on your versions of Windows and possibly of AD neither of which you stated.

Your Java6 attempt to host1 clearly offered at least one suite using ECDH that the server chose, but which failed to work in the Sun PKCS11 provider. I'm not certain about Solaris but the Windows and Linux Java6 distros from Sun did not include a builtin ECC provider and thus would not offer (or agree) any ECDH suite unless PKCS11 gives access to something that (allegedly) supported it. Java7 and 8 do come with ECC builtin, but if you have PKCS11 at provider position 1 it will still get preference.

Your openssl s_client negotiated a plain-RSA suite on host1. (The OpenSSL name AES256-SHA actually means TLS_RSA_WITH_AES_256_CBC_SHA; for historical reasons the OpenSSL names omit the RSA for plain-RSA key-exchange.) OpenSSL versions 0.9.* do not support ECC by default and would produce this result. (Technically you can get 0.9.8 to do ECC if you beat it enough, but I assume you would have mentioned that.) OpenSSL (upstream) 1.0.0 or greater supports all standard ECDH suites and prefers them, which I would expect to cause the server to select one. I know RedHat packages of OpenSSL omitted ECC until recently over patent concerns; obviously RedHat wouldn't apply on Solaris, but conceivably the (a?) Solaris packager might have done the same thing. It's also possible somebody tweaked your build to support ECC but not prefer it, and the server followed that preference. You might try adding -cipher ECDH:!aNULL and see if that succeeds in getting some ECDH suite and if so which.

Trusting wildly with Java: In any case, with openssl you don't need InstallCert. Do

echo Q | openssl s_client -connect $host:$port -showcerts \
| nawk '/--BEG/,/--END/{print>"anchor"} /--END/{close("anchor")}'

i.e. add -showcerts and select the last block of certificate data into a file, then

 keytool -keystore $YOURJRE/lib/security/cacerts -importcert -file anchor -alias $NAME

where $NAME is a brief name of letters and digits only mnemonic of your local authority (domain or whatever), and enter the password changeit and confirm.

Answer 2

What is the purpose of Java security providers and what is their role in encryption?

Excerpt from my jdk/jre/lib/security/java.security file:

#
# This is the "master security properties file".
#
# In this file, various security properties are set for use by
# java.security classes. This is where users can statically register
# Cryptography Package Providers ("providers" for short). The term
# "provider" refers to a package or set of packages that supply a
# concrete implementation of a subset of the cryptography aspects of
# the Java Security API. A provider may, for example, implement one or
# more digital signature algorithms or message digest algorithms.
#
# Each provider must implement a subclass of the Provider class.
# To register a provider in this master security properties file,
# specify the Provider subclass name and priority in the format
#
#    security.provider.<n>=<className>
#
# This declares a provider, and specifies its preference
# order n. The preference order is the order in which providers are
# searched for requested algorithms (when no specific provider is
# requested). The order is 1-based; 1 is the most preferred, followed
# by 2, and so on.
#
# <className> must specify the subclass of the Provider class whose
# constructor sets the values of various properties that are required
# for the Java Security API to look up the algorithms or other
# facilities implemented by the provider.
#
# There must be at least one provider specification in java.security.
# There is a default provider that comes standard with the JDK. It
# is called the "SUN" provider, and its Provider subclass
# named Sun appears in the sun.security.provider package. Thus, the
# "SUN" provider is registered via the following:
#
#    security.provider.1=sun.security.provider.Sun
#
# (The number 1 is used for the default provider.)
#
# Note: Providers can be dynamically registered instead by calls to
# either the addProvider or insertProviderAt method in the Security
# class.

[EDITED] I'll amend my answer to just the one question I'm qualified to answer here, ceding to dave_thompson's answer in regards to the cause of the bug.