Buffer overflows aren't detected at compile time. There are code analysis tools such as Sparse or Lint (cpplint, pc-lint) that will perform further analysis on both source code files or compiled binaries. Each analysis tool has their own algorithms for determining a buffer overflow, but it comes down to common known instructions that lead to buffer overflows.
You can also add Bounds Checking at compile which inserts bounds information for each allocated block of memory. This bounds information is then checked at run-time to ensure buffers are within their limits. A common implementation is to use "fat" pointers. Which contain both the address of the real pointer to the data, and additional data describing the region it is located. I believe Firefox does this for their memory allocations, but I could be mistaken.
Canaries are inserted at compile time to help detect buffer overflows by inserting a
word of data between a buffer and the control data on the stack. At a certain point before the return of the function the canary is verified to be intact.
ASLR has nothing to do with stack protection. It randomizes the address that your program runs in memory. This means that you can't rely on functions to be at the same address each time the program is run. This prevents the hardcoding of library and function addresses. Making the stack or heap (or any piece of memory that you're trying to execute) executable is necessary regardless, but ASLR will still cause you problems. If you're just experimenting and trying to understand exploit code I would do the following:
- Disable ASLR
- Disable stack protections
- Attack each problem individually, and re-enable them one at time until your exploit can handle both.
If you have a binary that crashes due to... well anything (not just a buffer overflow) run the program in a debugger. The debugger will catch the crash at the exact point that the program fails. You should probably realize that this might not be the exact location of the overflow, but a stack trace should help determine the root cause. I would suggest reading these posts if you're not familiar with reverse engineering tools, or x86 computer architecture.
Why are buffer overflows executed in the direction they are?
Reverse Engineering Tools