That is an interesting question, but it can be read at 2 different levels.
How can I prove that the owner of that key/X509 certificate is M. X?
Well you cannot... unless you have procedures that prove it. For example for x509 user certificates, there is often a face to face operation where the signer controls the identity of the recipient, and has him manually sign a document claiming that he received a certificate for a key he (and only he) owns from Y the day of month, year. Now you can prove (even to a legal court) that the owner of the key of the cert is M. X . But this assumes that you cannot know the key.
What can the admin. of a system do on a key file?
Unfortunately plenty of things! A admin can rewrite the file, put a new key in it, change the dates (access, modification and creation) of the file. The only thing that an admin cannot do is to read the content of an encrypted file if the secret is not in the system.
That means that if you want to be able to prove that a user is the perpetrator of an action, you will have to setup a procedure that guarantees that you cannot know the private key, but know the public key (x509 certificates can be a nice solution). But the key should never exist in decrypted form on your machine, because you as the admin could get it even from a process memory, or via system call hooks.
That is the reason why it is often said that a user should not use a machine he does not trust, and he cannot trust a machine if he cannot trust the administrator. Said differently, you will never find a purely technical way to prove that a user did perpetrate an action if the key lies in a disk on which you have administrator priviledges. The solution here is legal and not technical. The only technical way requires that the key is only installed on the user's machine and not on yours.
BTW the secret question way is also a dead end. The user could say: yes I did answer this to that question, but it was about that other action... Once he has given the secret answer to you, the secret is no longer his own but is shared between him and you.