Your best bet is to install a HIPS (Host Intrusion Prevention Application) such as Samhain or AIDE. There are far too many files to monitor and an attacker can (and usually will) try to modify whatever they can. Not to mention there are plenty of people who play/tinker with Linux/BSD/OSX viruses, exploits, and proofs of concepts
EDITED FOR AN EXPLANATION AS TO WHAT TO MONITOR AND WHY
Every system differs, one can follow guidelines, baselines, but at the end of the day, the importance of what to monitor as at the discretion of the systems administrator, the data custodian, and a range of other people, or just one person. On any system I have been an administrator on, it is my function to maintain that system. I prefer to know as much as I can about the entire system.
When I install Host Based Intrusion Prevention, I choose to monitor everything EXCEPT log files. Logfiles change, therefore they'd generate many false positives. As an attacker/pentester, I am aware that many professionals who've followed best practices and guidelines, tend to focus on what they perceive (based on risk) as important. This will usually (and mostly ONLY) include directories where binaries are installed. Under Linux/BSD/Solaris:
These are not the only places to store files, to hide programs, etc. For example, knowing these are the "usual suspects" being monitored, I have no time as an attacker shoving something into /usr/lib, /usr/share, /tmp, and so forth. These files are not going to be detected because no application is watching what is occurring.
The time and space to monitor these files/directories is minimal. I would rather be safe than sorry. There may be false positives initially, but this is where you can ween out what are legitimate alerts, and what aren't, building from there.