The NSS SSL Keylog file is a non-obtrusive way to extract SSL session keys from an application using the NSS library for SSL/TLS, but there is no standard way to do the same in all applications. (A generic approach would involve a man in the middle attack, but this is not really non-obtrusive.)
The idea of the extracting session keys directly from the application is more widely applicable though. libssl.so suggests that OpenSSL is in use on a (likely) Linux platform. You can either use use
LD_PRELOAD to interpose the OpenSSL library (may not work in all cases) or attach a debugger with appropriate breakpoints. Both approaches are documented and explored further in Extract pre-master keys from an OpenSSL application.
As an additional tip for troubleshooting, while Wireshark 1.12 is able to decrypt the TLS application data (and Follow SSL stream can be used to look at the communication), it does not dissect the SMTP traffic which occurs after a STARTTLS handshake (SMTPS is unaffected). In Wireshark 2.0 (tentative release date in November 2015, pre-release versions are available) the SMTP protocol is correctly recognized after STARTTLS.