From your description, it sounds like the server is currently using the key, which means the server "knows" the pass phrase. If this is correct and you have appropriate access to the server, you should be able to extract it. How you'd do that depends on what the server software is and how it's set up.
Just as an example, if you were running Apache, and it had something like this in the httpd.conf file:
That means that Apache will run /etc/apache2/getsslpassphrase to get passphrases; and you can do the same:
sudo /etc/apache2/getsslpassphrase server.example.com:443 RSA
should output the passphrase for the server.example.com key.
For other server software (or Apache with different config options), you'd have to specify the details.