I am learning about return to system call attacks for a security class. I understand that in this kind of attack, attackers replace the standard return value for a stack frame with the address of a library function. The slides from class say that this is a way to evade a non-executable stack. How so? Is the idea that the library function has been tampered with and so when the frame returns it goes to the malicious library code? Is a return to system call attack only useful if the library function has already been corrupted?
When a function exits, execution jumps to the address which was saved on the stack. "Classical" exploits of a buffer overflow alter that saved address so that it points to, precisely, the contents of the stack buffer which was just overflown. This assumes that the attacker gets to choose the contents of the buffer, and places there the code he wishes to get executed.
When the stack is non-executable, the classic exploit no longer works. However, the buffer can still be filled, and overflow over the rest of the stack, which includes the return address, but also other things. Consider that the stack looks like this:
In this rough diagram, the stack grows downwards (as it happens in most architectures); addresses are fictitious but realistic for a 32-bit little-endian architecture on a Unix-like system. The attacker will try, by inserting maliciously crafted data in the buffer, to obtain this situation:
If the attacker can get this, then, when the attacked function returns, the stack pointer is set to 0x07fff8b4, then one word is popped from the stack, and interpreted as the "return address", i.e. the point where execution shall continue. In this case, this makes the CPU jump to the standard library function
The code of
In our situation,
Summary: a buffer overflow which overruns the return address is used to make the attacked code jump to an arbitrary place. The attacker wants the CPU to run some code which gives the attacker some advantage (e.g. code which runs a shell). Since the stack is non-executable, the attacker cannot put his code in the stack. But the standard library is a collection of functions which are, by nature, executable, and some may do just what the attacker wishes to obtain. It then suffices for the attacker to make the return address point to the library function which corresponds to the desired effect, and the same buffer overflow can be made to place on the stack the "arguments" that the library function will use, right at the place where that function expects them. Since none of this uses the stack area as code, only as data, the non-executable property of the stack does not matter.
With an attack to the non-execute stack you wouldn't be able to inject malicious code with e.g. a buffer overflow exploit, that would obviously need to be done on the execute stack for it to be of any use. However, you'd still be able to replace the address in the call stack to execute any existing system libraries' functions. That's why these return to system call attacks are most commonly named return-to-libc attacks with