1. I have compiled openssl 1.0.1u (yes, i know its old one) with my custom ciphersuite (ECDHE-RSA-MYCIPHER-CBC-SHA) with openssl for a research project; it has compiled and installed successfully.
  2. When I try to run the command:

    "s_client -cipher'ECDHE-RSA-MYCIPHER-CBC-SHA' -connect localhost:443"


    "s_client-cipher 'ECDHE-RSA-MYCIPHER-CBC-SHA' -connect localhost:465"

    it successfully performs the handshake [wireshark capture shows]

  3. But when I try to run the command

    "s_client -cipher'ECDHE-RSA-MYCIPHER-CBC-SHA' -connect localhost: 993"

    it fails, handshake failure [Wireshark capture shows]

  4. In my current scenario, I am using the open source zimbra email server that is running nginx reverse proxy for the ports of HTTPS[443], SMTPS[465] and IMAPS[993]. The Ubuntu 14.04 lts openssl is not modified but the email server's openssl binary is modified.

  5. The nginx.log file shows the error to be

    SSL_do_handshake() failed (SSL: error:1408A0C1:SSL routines:SSL3_GET_CLIENT_HELLO:no shared cipher) while SSL handshaking

  6. What is strange is that Nginx proxy is running for the 3 ports mentioned above but handshake fails for the port 993 only [also it fails for 995] ? What am I missing or doing wrong?

Update 01 - Settings/Conf files of NGINX

File 01 --- /opt/zimbra/conf/nginx/includes/nginx.conf.mail

    # mail proxy connection timeout
    proxy_ctimeout              120000ms;

    # pass error messages from the backend server to the client
    # if true, the error messages are passed to the client verbatim, else
    # nginx logs the original error message in its log file and sends back
    # a generic error message to the client
    proxy_pass_error_message    on;

    # HTTP lookup handlers that will return the route information for a
    # pop3/imap login

    # Wether to use ssl to connect the upstream mail servers
    proxy_ssl on;

    # IMAP/POP3 greeting messages
    imap_greeting            "";
    pop3_greeting            "";

    pop3_capabilities         "EXPIRE 31 USER" "TOP" "UIDL" "USER" "XOIP";

    # IMAP4 ID Extension support (RFC 2971)
    # Use the imap_id directive to specify the string that should be sent
    # back by the proxy server back to the client upon receipt of an 
    # IMAP ID command as described by RFC 2971
    # There MUST be an even number of strings specified against 
    # this directive, because RFC 2971 defines the server response to be 
    # a list of field-value pairs (refer Formal Syntax, RFC 2971 Section 4)
    # Each string in this list should be enclosed within double quotes
    # If not quoted, they will be automatically quoted by nginx, but the 
    # use of quotes allows the space character to be included within an 
    # ID field or a value
    # The odd numbered strings are treated as ID fields, with the following 
    # even-numbered string considered to be the ID value corresponding to 
    # the field name
    # If omitted, the response to the ID command will be nil, which
    # according to RFC 2971, is permissible in lieu of the field-value list
    # The imap_id directive may be overridden in a server block
    imap_id         "NAME" "Zimbra" ;

    # Default realm (kerberos)
    # For GSSAPI authentication, when the server's realm is the same as 
    # the default realm as defined in the default_realm variable in the 
    # [libdefaults] section of krb5.conf, then in this case, the SASL 
    # GSSAPI library strips off the realm portion from the authenticating
    # principal. Therefore, the default realm name must be specified here
    # so that NGINX can append it to the authenticating principal for GSSAPI
    default_realm           "";

    # Resolve IP address to SASL server name (kerberos)
    # For GSSAPI, in cases of multi-homed hosts, it may be that the proxy
    # server has multiple network interfaces, and we wish to reverse map 
    # the incoming interface IP address (via DNS) to a host name, which will
    # be used as the service principal
    # Set this to on if gssapi clients will connect to a proxy server using
    # different FQDNs. Otherwise, set it to off, and the fully qualified 
    # host name of the proxy (as returned by hostname), will be used as the
    # service principal
    # If not specified, this configuration defaults to off
    sasl_host_from_ip       off;

    # sasl_app_name
    # This is the application name which nginx will use when initializing
    # the SASL library using the call to sasl_server_init()
    # The SASL library is initialized once per process, the application name
    # provided here is used for where to find the default configuration file
    # If not specified, sasl_app_name defaults to "nginx"
    sasl_app_name           "nginx";

    # Login rate limiting directives

    # mail_login_ip_max 
    # Sets the maximum number of times that any user is allowed to log in from
    # a particular IP over POP or IMAP to this proxy server before the login is
    # rejected with an appropriate protocol specific bye response
    # This counter is cumulative for all users that appear to the proxy to be 
    # logging in from the same IP address
    # If multiple users appear to the proxy to be logging in from the same IP 
    # address (usual with NATing), then each of the different users' login will
    # contribute to increasing the hit counter for that IP address, and when the
    # counter eventually exceeds mail_login_ip_max, then the connections from 
    # that IP address will be throttled
    # Therefore, all users from the same IP will contribute to
    # (and be affected by) this counter
    # Logins using all protocols (POP3/POP3S/IMAP/IMAPS) will affect this 
    # counter, (the counter is aggregate for all protocols, *not* separate)
    # See notes accompanying the mail_login_ip_ttl for reasonable values for
    # this directive
    # If this value is set to 0, then no throttling will take place for any IP
    mail_login_ip_max               0;

    # mail_login_ip_ttl
    # Sets the time-to-live for the hit counter for IP login throttling
    # Used in conjunction with mail_login_ip_max, this defines the semantics
    # of the throttling for IP logins. Therfore, if this is set to 3600s, and
    # if mail_login_ip_max is set to 1000, then it means that NGINX should not
    # allow more than 1000 users to log in to the proxy from the same IP, 
    # within the time interval of an hour (=3600s)
    # The semantics for such a configuration would then be -
    # "allow maximum 1000 users per hour from any given IP address"
    mail_login_ip_ttl               3600000ms;

    # Define a textual message that should be displayed to the user when
    # his/her connection is rejected based on IP overusage
    # This message will be encapsulated in the proper protocol specific 
    # response to the client ("* BYE" for IMAP, and "-ERR" for POP3)
    mail_login_ip_rejectmsg         "Login rejected from this IP";

    # Similar semantics for Rate Limiting User Logins
    # Setting mail_login_user_max to 100 and mail_login_user_ttl to 3600s
    # implies "allow maximum 100 logins per hour for any user"
    # As with the ip counterparts, the user hit counter and timeout are 
    # cumulative for all protocols
    # Also, for a given user's login, both counters are checked in succession,
    # with the IP counter being checked first
    # A login may be rejected (throttled) because the IP is over-usage, or
    # because the login name itself is over-usage
    # A value of 0 indicates that no throttling will take place for any user
    mail_login_user_max             0;
    mail_login_user_ttl             3600000ms;
    mail_login_user_rejectmsg       "Login rejected for this user";

    # Issue POP3 XOIP before logging in to upstream (Audit purposes)
    proxy_issue_pop3_xoip   on;

    # Issue IMAP ID before logging in to upstream (Audit purposes)
    proxy_issue_imap_id     on;

    # Supported SASL Authentication mechanisms
    # Use the pop3_auth and imap_auth to specify which SASL mechanisms are 
    # enabled for POP and IMAP respectively
    # These directives may be specified at mail{} level, or overridden at
    # server{} level
    # Even though an authentication mechanism may be listed against 
    # pop3_auth and/or imap_auth, it may or may not be available depending
    # on whether cleartext login is available (see description for starttls)
    # For example, if starttls is set to only, then SASL PLAIN is not 
    # available outside of TLS/SSL
    # SASL mechanisms may be specified all on one line (space separated)
    # against pop3_auth, or they may be specified on a separate line
    # Specifying the SASL mechanisms on a separate line has the advantage
    # that they can be enabled/disabled separately by zmmtaconfig using
    # different LDAP attributes
    # Currently, only PLAIN and GSSAPI mechanisms are supported

    imap_auth         plain;
    #imap_auth         gssapi;

    pop3_auth         plain;
    #pop3_auth         gssapi;

    # Specify whether to use literal strings while issuing the LOGIN command
    # to the upstream server. If set to on, then it means that NGINX will use
    # literal strings to log in to the upstream server, and so the upstream 
    # MUST support literal strings
    # If set to off, then NGINX uses quoted strings to log in to the upstream
    # server when using the LOGIN command
    # If not specified, imap_literalauth defaults to on
    imap_literalauth        on;

    # Auth Wait Interval
    # Specifies the time interval that NGINX will wait before rejecting an
    # authentication attempt to the upstream mail server with invalid credentials.
    # This value is not related to the wait time when Zimbra lookup handler replies
    # some login error, which is controled by the "Auth-Wait" header returned by the
    # lookup handler.
    # If not specified, this value defaults to 10 seconds.
    zm_auth_wait               10000ms;

    # TLS configuration
    ssl_prefer_server_ciphers   on;
    ssl_certificate     /opt/zimbra/conf/nginx.crt;
    ssl_certificate_key /opt/zimbra/conf/nginx.key;

    # SSL Protocols & Ciphers configuration
    # Enabled protocols & Permitted ciphers. Ciphers are assigned in the formats supported by OpenSSL
    ssl_protocols            TLSv1 TLSv1.1 TLSv1.2;

    # SSL ECDH cipher curve configuration
    ssl_ecdh_curve          prime256v1;

     include /opt/zimbra/conf/nginx/includes/nginx.conf.mail.imap;
     include /opt/zimbra/conf/nginx/includes/nginx.conf.mail.imap.default;
     include /opt/zimbra/conf/nginx/includes/nginx.conf.mail.imaps;
     include /opt/zimbra/conf/nginx/includes/nginx.conf.mail.imaps.default;
     include /opt/zimbra/conf/nginx/includes/nginx.conf.mail.pop3;
     include /opt/zimbra/conf/nginx/includes/nginx.conf.mail.pop3.default;
     include /opt/zimbra/conf/nginx/includes/nginx.conf.mail.pop3s;
     include /opt/zimbra/conf/nginx/includes/nginx.conf.mail.pop3s.default;

File 02 --- /opt/zimbra/conf/nginx/includes/nginx.conf.mail.imaps.default

# IMAPS proxy default configuration
    #listen                  [::]:993;
    listen                993;
    #listen                [::]:993 ipv6only=on;
    protocol            imap;
    proxy               on;
    timeout             60;
    proxy_timeout       2100;
    ssl                 on;
    ssl_certificate     /opt/zimbra/conf/nginx.crt;
    ssl_certificate_key /opt/zimbra/conf/nginx.key;
    sasl_service_name   "imap";

Update 02 - Updated System's (ubuntu14.04 lts x64 openssl binary)

So, I know have replaced the Ubuntu's 14.04 LTS x64 openssl binary with my openssl binary (containing the custom cipher). I renamed the original openssl binary in the '/usr/bin/openssl' to opensslORIG and then 'mv' custom cipher containing openssl binary in the '/usr/bin/openssl' ... but the exact same problem/error is still there.

Update 03 - Issue is reolved

I updated the open source email server (newer version); it also updated the nginx and other modules. Then I applied the same steps for integration of custom cipher in the openSSL. Now, everything is working properly (custom cipher on all ports). So, the issue was of the EMAIL SERVER not of nginx.

  • 1
    Please provide the configuration of ngnix. My guess is that port 993 is configured with protocol imap and thus is endpoint for the SSL connection while the other ports are configured with protocol tcp and thus pass the SSL connection through. Also I've removed that tag java since nothing in your question indicates any use of Java. – Steffen Ullrich Jan 9 '18 at 15:38
  • the zimbra uses jetty, i was wondering if somehow java has some kind of setting / configuration required. – Fuzed Mass Jan 9 '18 at 15:45

The configuration shows that you have configured nginx as the SSL endpoint, i.e. it will terminate the SSL connection with the given certificates. nginx uses the systems OpenSSL library which does not include your cipher. Since this is the only cipher your client offers the connection will fail because of no shared ciphers.

  • if i provide nginx to use the different openssl (to use the one which i have compiled) binary will it work ? or what should be the best approach in this scenario ? – Fuzed Mass Jan 10 '18 at 1:45
  • so, I know have replaced the Ubuntu's 14.04 LTS x64 openssl binary with my openssl binary (containing the custom cipher). I renamed the original openssl binary in the '/usr/bin/openssl' and then 'mv' my openssl binary... but the exact same problem/error is still there. – Fuzed Mass Jan 10 '18 at 3:51
  • @FuzedMass: nginx does not use the openssl binary. It uses the openssl library, i.e. libssl and libcrypto (check ldd /usr/bin/nginx for details).It might work if you replace the libraries depending on how far away your library is from the system library. But you might also render the system unusable due to this since it might be that the libraries don't provide the same API. If you want to keep the configuration of nginx being the TLS endpoint you better recompile nginx against your changed libraries. – Steffen Ullrich Jan 10 '18 at 6:33
  • thanks for the detailed response once again. I ran the "ldd" command the nginx is already using the modified OPENSSL libraries "libssl.so" & "libcrypto.so". The path to those library is of the modified openssl binary and library. what do you suggest now ? – Fuzed Mass Jan 10 '18 at 6:45
  • 1
    @FuzedMass: the next thing is that you need to have the ciphers enabled in nginx so that it is actually considered by the server. See ssl_ciphers parameter. – Steffen Ullrich Jan 10 '18 at 6:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.