Extracting openssl pre-master secret from nginx

Question

I have to store pre-master keys for all TLS connections due to regulator request. We are using nginx to terminate TLS.

I've read Extract pre-master keys from an OpenSSL application and especially excellent walkthrough Extracting openssl pre-master secret from apache2 and I was able to record keys from apache, but still have no success with nginx. Perhaps I'm missing something obvious?

I'm using Debian 9 system for testing. First I check which version of openssl is used by nginx:

# nginx -V 2>&1 |grep SSL
built with OpenSSL 1.1.0k  28 May 2019
# ldd /usr/sbin/nginx |grep ssl
    libssl.so.1.1 => /usr/lib/x86_64-linux-gnu/libssl.so.1.1 (0x00007f9bd9f95000)

So I installed libssl-dev package which provides 1.1.0k-1~deb9u1 version.

I downloaded sslkeylog.c and compiled it: cc sslkeylog.c -shared -o libsslkeylog.so -fPIC -ldl

I put compiled library into /usr/local/lib/libsslkeylog.so path for convenience. Then I edited nginx's systemd config:

#cat /etc/systemd/system/nginx.service.d/override.conf
[Service]
Environment=SSLKEYLOGFILE=/tmp/premaster.txt
Environment=LD_PRELOAD=/usr/local/lib/libsslkeylog.so

# systemctl daemon-reload
# systemctl restart nginx

I can see that LD_PRELOAD works - lsof of both master and worker nginx processes show that /usr/local/lib/libsslkeylog.so is loaded:

# lsof -n -p 10313 |grep ssl
nginx   10313 root  mem    REG              254,1   442984   3255 /usr/lib/x86_64-linux-gnu/libssl.so.1.1
nginx   10313 root  mem    REG              254,1    14224  20914 /usr/local/lib/libsslkeylog.so
# lsof -n -p 10314 |grep ssl
nginx   10314 www-data  mem       REG              254,1   442984   3255 /usr/lib/x86_64-linux-gnu/libssl.so.1.1
nginx   10314 www-data  mem       REG              254,1    14224  20914 /usr/local/lib/libsslkeylog.so

But after I access nginx through curl

curl -k -I https://localhost

or through browser on my PC, /tmp/premaster.txt is not created

What am I doing wrong? Or is there a better way to store pre-master keys in nginx?

Answer 1

Well, as I was pointed at nginx mailing list, nginx removes all environment variables inherited from its parent process except the TZ variable, so once I defined needed variables in nginx.conf

env LD_PRELOAD=/usr/local/lib/libsslkeylog.so;
env SSLKEYLOGFILE=/tmp/premaster.txt;

keys started being recorded as expected.

Answer 2

The answer above seems to be incorrect. nginx doesn't execve() its children, those are just forks of the parent process, so setenv() of LD_PRELOAD doesn't have any effect, it needs to be set before nginx is executed at all, i.e. in SystemD.

However SSLKEYLOGFILE does need to be passed to workers.

So, my advice is to follow the same steps as for Apache, but with the added step of putting env SSLKEYLOGFILE; in nginx.conf.

Summary of the steps needed:

• compile libsslkeylog.so as usual:

  sudo apt install git make gcc libssl-dev
  git clone --depth=1 https://git.lekensteyn.nl/peter/wireshark-notes
  cd wireshark-notes/src
  make
  sudo install libsslkeylog.so /usr/local/lib/
  

• set environment variables in systemd, again as usual:

  cat > /etc/systemd/system/nginx.service.d/override.conf <<EOF
  [Service]
  Environment=SSLKEYLOGFILE=/tmp/premaster.txt
  Environment=LD_PRELOAD=/usr/local/lib/libsslkeylog.so
  EOF
  

• make sure SSLKEYLOGFILE remains accessible to workers, by adding the following to nginx.conf:

  env SSLKEYLOGFILE;
  

Update: It seems this is still not enough for nginx to log all secrets. There's an undocumented OpenSSL function called SSL_set_SSL_CTX which replaces the context of an SSL object. nginx uses this function. libsslkeylog is not prepared for this (it only taps SSL_new and registers keylog on that context). I'll try to contact the author (@lekensteyn) to improve it.

Update 2: If your version of nginx has HTTP/3 support enabled, check whether it was compiled with a fork of OpenSSL that supports HTTP/3, or if it's using the so called "OpenSSL compatibility layer", which is a hack that allows nginx to serve HTTP/3 using vanilla OpenSSL. This hack uses the keylog callback to make it work, so it will conflict with libkeylogssl. To make libkeylogssl work with the hack, some elaborate scheme is needed to enable its keylog callback to work in addition to the one set by the application.