If C uses shared libraries, is it possible to write a program that will modify the fopen function to execute a malicious code? How can the OS protect the user from this?

2 Answers 2


Yes, that is possible. This technique is called function hooking. In fact, you can completely control an executable by hooking, say, __libc_start_main and redirecting execution to code in the library. There is absolutely no way an OS can protect an executable from a malicious library. The code in the library will run in the same address space as the executable and will have the exact same abilities. The only way to protect an executable from a malicious library is to not link it with malicious libraries.

  • The interesting about that is: antivirus can't get that from my understand, right?
    – Faminha102
    Commented Oct 30, 2018 at 2:00
  • @Faminha102 Antivirus can't get what? If you start antivirus from a compromised environment, then you could load a malicious library with the AV, but generally it is started from a privileged location (e.g. as a service), which means that you'd need to already have high privileges to provide it with a malicious library.
    – forest
    Commented Oct 30, 2018 at 2:15
  • 1
    @forest I assume OP meant that an Antivirus can't detect that you're trying to make some innocent program load a malicious library and thus execute malicious code. Probably OP is trying to put it into action. Commented Oct 30, 2018 at 10:27
  • @SombreroChicken Ah. Well AV will definitely try to scan the library, but obviously you could avoid AV detection in libraries just as you can in executables.
    – forest
    Commented Oct 31, 2018 at 1:49

Also, to be clear: when you modify a shared library in memory, only the copy of it in your process' address space gets modified. Neither the copy on the disk nor the copies in use by other processes will get modified. It's the same as modifying a global variable in a process (such as errno); only the one process using that address space will "see" the modification.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .