As with many things, ensuring you detect a successful exploit requires defense in depth, and a little preparation.
We'll start with the assumption that an attacker will somehow break the "target software" and either have it or the account it runs under execute arbitrary code supplied by the attacker. Our goal is to be able to detect that this has occurred.
First, on the prevention side, you'll want to ensure that whatever target software runs under a separate account with least privileges. This implies that even a successful attack will not have access to the rest of the system. You'll also want to make sure the system gets regular security updates, etc. This mitigates the risk that the attacker will be able to elevate privileges from the account he popped to root or something else interesting. Finally, have only the minimal software needed, ideally only for a single purpose. (Don't comingle web/app/database on the same box.)
Now once you have those done, you can start looking detecting the exploit. For that, you should start with OS event logging. Did any unusual processes start or stop. Presumably, there'll be a pattern of "normal" usage, and if the attacker runs their own code, it might be in a separate process. An IPS/IDS should provide lots of insight into this -- much better to find a good one and configure it than to try to roll it yourself. Any IPS/IDS worth its salt should be sending event data out to a separate host (as @user400344 suggests).
On the OS firewall side, most server software doesn't initiate TCP connections to random endpoints, so you may want to configure the firewall to only allow traffic outbound to known destinations. This would also make it a little harder for an attacker to exfiltrate data. Upstream from the server, you could monitor TCP traffic to ensure it matches what you expect.
Keep a log of all processes started. There are various programs that can help you do this. (Also pipe that off the host.)
Next, at the application layer, you'll want to write to syslog or another service that runs separate from the target software. During the exploit, the attacker would be able to write to syslog, but not modify it.
If the target software schedules jobs, accepts commands, etc., then check to ensure those aren't modified except through an expected workflow.
You could run an IPS/IDS at the application layer too (web app firewall, if the target software is a web server), in which case that should provide additional defense.