If C uses shared libraries, is it possible to write a program that will modify the
fopen function to execute a malicious code? How can the OS protect the user from this?
Yes, that is possible. This technique is called function hooking. In fact, you can completely control an executable by hooking, say,
__libc_start_main and redirecting execution to code in the library. There is absolutely no way an OS can protect an executable from a malicious library. The code in the library will run in the same address space as the executable and will have the exact same abilities. The only way to protect an executable from a malicious library is to not link it with malicious libraries.
Also, to be clear: when you modify a shared library in memory, only the copy of it in your process' address space gets modified. Neither the copy on the disk nor the copies in use by other processes will get modified. It's the same as modifying a global variable in a process (such as
errno); only the one process using that address space will "see" the modification.