If we have a program that contains a secret key in its compiled code, is it possible to write a program that can automatically extract the secret key? Let's assume that the program contains a secret key is not obfuscated and the secret key is used to decrypt some messages. We can know the extracted key is correct if we can use it to decrypt messages.
To some degree, yes. An executable file might look like gobbledygook to the uninitiated, but it's actually a structured file. There are various tools you can use to inspect an executable and dump the executable into various sections that you can analyse further.
In the simplest case, you'd simply dump the .rodata section of an ELF executable and then the key might be somewhere there. If the key have a recognisable header1, you can simply scan for that key header and try that as your key. In the slightly more complex cases, you may have to write a code to look into the symbol table to disassemble some functions to get the address and/or size of the key. In the worst case, you'll just have to try every offset into the .rodata or the executable file one by one until you successfully decrypted the message. In some more interesting case, the target program might have a function to retrieve the key, so the key extractor program can simply just load the executable like a library, lookup that function's address in the symbol table, and execute that function to extract the key. Another possible technique is that the analyser might just run the target program in a debugging harness, set a breakpoint, and capture the values in memory as the program loads the key or passess the key to the decryption function.
If you know roughly how the target executable is structured, it's not too difficult to write an automatic analyser that can automatically extract various data from the executable, even ones that can tolerate as things move around a bit throughout various updates of the target program. Depending on how robust the analysis is, it may be possible to extract that data from a large class of programs that uses known/similar strategy to store the key. People have used very similar techniques to write game patches/cheat engines/trainers.
However, if your question is whether it's possible, in the general case, to write a fully automated key extractor that will always work for arbitrary executable that you throw at it, even ones that uses novel approaches to store the key that hasn't been designed into the analyser and with zero prior human analysis of the approach. Then the answer is /probably/ not, due to the Halting problem; static analysis can never fully predict arbitrary running programs' behaviour. Though in practice, if the key isn't obfuscated at all in the executable, then the brute force approach of just trying every offset of the executable would have quite a good chance of success.
1 For example, an ASCII-armoured PGP key always begins with
-----BEGIN PGP PRIVATE KEY BLOCK-----. Even the binary encoded version of binary data structures might have magic numbers that can be used to help identify it.
This depends on how is manage the private key on the binary, and probably you will need to do reverse engineer to figure out the correct private key. I have seen keys split-ed in two parts: one on the text segment of the binary and the other component was opcodes on the execution segment as well as some keys can make mathematical operations on the assembler to be tricky to discover.
By the way, ask this question on https://reverseengineering.stackexchange.com/ probably you will find better responses