[Does] compiling your own binary for an application (with specific compilation flags) instead of using the "mainstream binary" make it more difficult for an attacker to leverage buffer overflows?
It depends on the operating system, the language the application is written in, and the compiler.
Second the programming language must allow manual memory management. Java and C# use automatic memory management, preventing the basic buffer overflow vulnerability. C and C++ allow manual management of dynamic memory. If the programming language has managed memory, then compiling will not help.
Third, the compiler or libraries used in the application must support some extended dynamic memory management monitoring or control. The Microsoft C++, Intel C++, GNU C++, LLVM Clang C++ compilers all support -fstack-protector, IBM XL C++ supports -qstackprotect. Some libraries like Avaya Labs's LibSafe, also have mechanisms to protect the stack. If the compiler doesn't support stack protection, and there are no drop in stack protection libraries available, then compiling will not help.
Fourth, the operating system may provide some protection. Some operating systems already protect the stack by exeuctable bits, stack base pointers, and saving return addresses in registers. If the operating system is already protecting the stack, the compilation will not help.
Is it way less efficient than Address space layout randomization ?
Address space randomization is a technique that prevents an attacker of using a well known address to call a system function. For example if setuid() is always at address 0xDEADBEEF then the attacker can attempt to overwrite a return address with 0xDEADBEEF and execute setuid. The randomization does not prevent buffer overflows, it just prevents the use of static address values.
Is this point rendered moot by new memory management strategies in OS kernels?
Not necessarily. It depends on the OS. Some OSes are more focused on performance than security. Those OSes may actually introduce more vulnerabilities with memory management tricks.