There are two places you could look for evidence of such an exploit: the syslog and coredumps.
System log
I'll assume you are talking about the record of the exploit attempt itself and nothing before it (such as remote access logs), and that you have not enabled any extra logging which may track file accesses or process creation. For the exploit itself, there is typically no trace left behind in log files if the attack succeeded. The only way there would be is if it failed at least once and triggered a segmentation fault, which may be logged to the kernel log buffer (shown via dmesg), which itself is likely logged by your choice of syslog daemon.
Core files
If you have coredumps enabled, then you may be able to get a lot more information about exactly what caused the segfault, assuming you have and know how to use gdb, and have the original binary which had crashed. as processes will log their core memory to the disk upon such a crash. But just like checking the system log, this only works if the exploit has failed at least once and triggered a crash. It also does not provide any information about who the attacker is, what their vector of exploitation was, what they are after, or what they used to attack. Also remember that, unless you have a special setup, coredumps are written to the disk with the same permissions as the program which crashed, so in some cases an attacker may have sufficient privileges to erase it (and examining that act goes into filesystem forensics, which I won't go into and isn't particularly related).
Compile-time hardening
As for what I do, I compile all programs locally, so I can use compile-time security features which attempt to trigger crashes on exploitation attempts. A few examples:
UBSAN (Undeined Behavior SANitizer) - Traps various types of undefined behavior to the ud2 instruction, which is an intentionally illegal instruction and crashes the program. Some types of undefined behavior can be used to exploit a program.
SSP (Stack Smashing Protector) - Creates a secret canary on the stack and aborts the program if it is ever overwritten. An attacker who only has an arbitrary write primitive will not be able to easily exploit a program which is utilizing SSP, unless they also have an arbitrary read primitive and can read the value of the canary (and put it in his overflow code).
FORTIFY_SOURCE - A collection of header files which wrap certain functions that expect a length as their argument, like strncpy(), to add runtime checks to ensure that the maximum length is not exceeded when it is known at compile-time.
PIE (Position Independent Executable) - Adds support to an executable for addressing libraries who's base is not known at compile-time. This allows a program to make use of ASLR, which loads libraries at random addresses to make it more difficult to exploit buffer overflows.
RELRO (RELocations Read Only) - In partial mode, makes ELF sections read-only after the linker has set everything up. In full mode, additionally makes the GOT read-only.
BINDNOW - Causes libraries to be resolved at executable load time, rather than when they are first referenced, which is necessary to benefit from the full potential of RELRO.
There are more security-related features which can be done at compile-time, like CFI and SafeStack. That was just a list of some of the most popular. You can use the updated checksec script to see whether or not a binary has bee compiled with various hardening flags:
$ checksec --file /bin/bash
RELRO STACK CANARY NX PIE RPATH RUNPATH FORTIFY Fortified Fortifiable FILE
Full RELRO Canary found NX enabled PIE enabled No RPATH No RUNPATH Yes 13 33 /bin/bash
Log entry examples
Logs will generally only be created when the program is compiled with the correct hardening flags (with the exception of segfaults which will occur regardless of how you compile the binary). These hardening flags do not guarantee that an attacker will be unable to exploit a given binary, but it does make it more likely that an exploit will fail and possibly leave entries in logs. Some examples:
Segfault triggering SIGSEGV (in this case, a null pointer dereference):
$ gcc <<< 'void main() { char *a = 0; *a = 5; }' -x c - 2>/dev/null
$ ./a.out
a.out[24078]: segfault at 0 ip 000000720ab0b6f4 sp 000003c8ad38d170 error 6 in a.out[720ab0b000+1000]
Segmentation fault
UBSAN triggering SIGILL (in this case, an out of bounds index):
$ gcc <<< 'void main() { char a[2]; a[-1] = 5; }' -x c - -fsanitize=undefined -fsanitize-undefined-trap-on-error 2>/dev/null
$ ./a.out
traps: a.out[24130] trap invalid opcode ip:62a7e64673 sp:3be53c5fe20 error:0 in a.out[62a7e64000+1000]
Illegal instruction
Stack Smashing Protector (written only to stderr):
$ gcc <<< 'void main() { char a[2]; strcpy(a, "abcd"); }' -x c - -fstack-protector-all -include string.h 2>/dev/null
$ ./a.out
*** stack smashing detected ***: ./a.out terminated
Killed
FORTIFY_SOURCE (written only to stderr):
$ gcc <<< 'void main() { char a[2]; strncpy(a, "abcd", 4); }' -x c - -O -D_FORTIFY_SOURCE=2 -include string.h 2>/dev/null
$ ./a.out
*** buffer overflow detected ***: ./a.out terminated
Killed
But again, and I can't stress this enough, a successful exploit will not leave logs like these. You must check for signs of the original intrusion and not rely on the chance that their exploit fails.
