I'm reading about stack overflow protection mechanism like DEP, and it's written that Ret2plt and
system() can be used to bypass it.
What is Ret2plt and how does it work? I cannot find a single explanation of this term on google.
In short, jump to the address of the PLT entry for the function = call the function.
Ret2plt is a similar concept to Ret2libc regarding DEP. DEP/Data Execution Protection works by making some memory non-executable (heap, stack..). This works to prevent code execution if the malicious code is smuggled into those non-executable areas. But if we could call code that has to be executable, like libc code, then it will be executable. At the assembly level we could even pass it our own parameters.
Described here: Rafal Wojtczuk, 1998, return to PLT uses the procedure linkage table to indirectly call libc functions.
The PLT is necessary because library code is not loaded into memory at fixed addresses, usually it is position-independent code (PIC). So procedure calls into shared libraries are made through the PLT. If we return into the PLT, then we are effectively calling those shared libraries linked by the PLT.
The main limitation is that you can only call functions that are already being used by the program, or else they won't have a link in the PLT.