how does a return to system allow an attacker to evade a non-executable stack?

Question

I am learning about return to system call attacks for a security class. I understand that in this kind of attack, attackers replace the standard return value for a stack frame with the address of a library function. The slides from class say that this is a way to evade a non-executable stack. How so? Is the idea that the library function has been tampered with and so when the frame returns it goes to the malicious library code? Is a return to system call attack only useful if the library function has already been corrupted?

Answer 1

When a function exits, execution jumps to the address which was saved on the stack. "Classical" exploits of a buffer overflow alter that saved address so that it points to, precisely, the contents of the stack buffer which was just overflown. This assumes that the attacker gets to choose the contents of the buffer, and places there the code he wishes to get executed.

When the stack is non-executable, the classic exploit no longer works. However, the buffer can still be filled, and overflow over the rest of the stack, which includes the return address, but also other things. Consider that the stack looks like this:

address     contents
0x07fff8c8  ...
0x07fff8c4  ...
0x07fff8c0  ...
0x07fff8bc  ...
0x07fff8b8  ...
0x07fff8b4  return address
0x07fff8b0  ...
0x07fff8ac  ...
0x07fff8a8  buffer
0x07fff8a4  buffer
0x07fff8a0  buffer
(...)
0x07fff858  buffer
0x07fff854  buffer

In this rough diagram, the stack grows downwards (as it happens in most architectures); addresses are fictitious but realistic for a 32-bit little-endian architecture on a Unix-like system. The attacker will try, by inserting maliciously crafted data in the buffer, to obtain this situation:

address     contents
0x07fff8c8  ...
0x07fff8c4  0
0x07fff8c0  0x07fff854
0x07fff8bc  0x07fff854
0x07fff8b8  ... 
0x07fff8b4  address of execlp()
0x07fff8b0  ...
0x07fff8ac  ...
0x07fff8a8  buffer
0x07fff8a4  buffer
0x07fff8a0  buffer
(...)
0x07fff858  0x0068732f
0x07fff854  0x6e69622f

If the attacker can get this, then, when the attacked function returns, the stack pointer is set to 0x07fff8b4, then one word is popped from the stack, and interpreted as the "return address", i.e. the point where execution shall continue. In this case, this makes the CPU jump to the standard library function execlp() (that's a Unix function). This function is, by definition, located in an area of RAM which is amenable to execution.

The code of execlp() expects to find the stack populated like this:

address     contents
0x07fff8c8  ...
0x07fff8c4  arg2
0x07fff8c0  arg1
0x07fff8bc  arg0
0x07fff8b8  address of calling code 

execlp() will interpret arg0 as a pointer to a zero-terminated character string which is the path of an executable file, which execlp() will run. arg1, arg2... shall be pointers to other strings which are the arguments to be passed to the executable file.

In our situation, execlp() has no idea how it was "called"; all it knows is that the CPU is now entering it. It finds as arg0 the value 0x07fff854 that the attacker arranged to put at this place on the stack, with his buffer overflow. That value, 0x07fff854, happens to be a pointer to the area of RAM which formerly contained the buffer, and still contains the data that the attacker placed in it because while this area is "free", the stack has not yet regrown into it. At address 0x07ff854, the attacker placed the two 32-bit words 0x6e69622f and 0x0068732f, which happen to be the encoding of the ASCII string "/bin/sh" (with a terminating 0).

Thus, execlp() finds properly formed arguments which instruct it to run a shell, and so it does.

Summary: a buffer overflow which overruns the return address is used to make the attacked code jump to an arbitrary place. The attacker wants the CPU to run some code which gives the attacker some advantage (e.g. code which runs a shell). Since the stack is non-executable, the attacker cannot put his code in the stack. But the standard library is a collection of functions which are, by nature, executable, and some may do just what the attacker wishes to obtain. It then suffices for the attacker to make the return address point to the library function which corresponds to the desired effect, and the same buffer overflow can be made to place on the stack the "arguments" that the library function will use, right at the place where that function expects them. Since none of this uses the stack area as code, only as data, the non-executable property of the stack does not matter.

Notes:

• There is no need to jump to the start of a library function. Jumping to the middle of a function also works. The possibilities are almost endless.

• Things work similarly for overflows which replace pointers to functions, e.g. vtables for C++ objects in the heap. What matters is that the target library code finds on the stack what it expects as arguments.

• Non-x86 architectures make things a bit more complex because the calling conventions for most RISC architectures (ARM, Sparc, PowerPC, Mips...) define that the first arguments are taken from conventional registers, not read from the stack. The basic principle remains valid, though, but it means that it is harder to make a one-size-fits-all exploit code. The attacker has to tune more details with regards to the exact application he attacks.

• The attacker must be able to predict, with good probability, where the library code will be loaded in the application address space. ASLR is a security feature which chooses the loading address more or less randomly, precisely to make such games harder for the attacker.

Answer 2

With an attack to the non-execute stack you wouldn't be able to inject malicious code with e.g. a buffer overflow exploit, that would obviously need to be done on the execute stack for it to be of any use. However, you'd still be able to replace the address in the call stack to execute any existing system libraries' functions. That's why these return to system call attacks are most commonly named return-to-libc attacks with libc being the C runtime library on UNIX systems. Imagine replacing the address of some function call with the address of libc's system function. This function only takes a single parameter, making it so much easier to replace, while being able to execute any preexisting function that would open-up the target system to further exploits. The answer to your question is thus no, you don't have to rely on having any malicious code installed on the target system, or that its libraries have to be already corrupted, when you already have the means to open up the target system by using its own preexisting function calls.