Your exploit payload ends up on the stack because you're overflowing a buffer on the stack, and this is how you gain control of the return address as well.
ESP points directly to the start of your payload (after execution of the
ret in the function you're attacking) because you put the payload right after the 4 bytes that overwrite the return address on the stack.
ret pops 4 (or 8) bytes into EIP, leaving ESP pointing to the payload that directly follows.
But you don't know what value ESP will have at that point, because of stack ASLR and because a different depth of call stack leading up to this point could change the address. So you can't hard-code a correct return address.
But if there are bytes that decode as
jmp esp or
call esp anywhere at a fixed (non-ASLRed) address in the process's memory, you can hard-code that address as the return address in your exploit. Execution will go there, then to your payload.
This is in fact often the case: Some DLLs don't have ASLR enabled for their code, and the main executable's code may not be ASLRed either.
Code ASLR for all code defeats a
jmp esp attack, unless the attacker can cause the target process to leak addresses.
Note that for 64-bit code, you're unlikely to be able to use
jmp rsp for string-based buffer overflows, because code addresses will contain some leading
jmp esp gives you a much more reliable exploit than repeatedly guessing a return address (with a very large NOP sled).
Repeated guessing will crash the target process every time you're wrong, but a
jmp esp can give you a high chance of success on the first try. This will avoid leaving crash logs. It could also defeat an intrusion-detection system that looks for crashing server processes and blocks connections from your IP address, or similar.
Note that the 2-byte instruction you're looking for can appear as part of another instruction when the program executes normally, or as static data (especially read-only data is often in executable pages). So you just need to search for the 2-byte sequence, not for
jmp esp in disassembly of the program. Compilers will never use
jmp esp, so you won't find one that way.
More generally, any function that ends with a buffer pointer in any register (e.g. from a
memcpy or especially
strcpy) can allow a ret2reg attack, by looking for a
jmp eax instruction.
This can work in 64-bit mode, where addresses have some high zero bytes; if
strcpy's trailing zero writes that high address byte for you, the end of your exploit string could be the non-zero address bytes that overwrite the return address on the stack.
In this case, the executable payload would go before the return address, at the spot in the buffer where the function leaves a register pointing. (Typically the beginning of the buffer if there are any useful pointers to the buffer in registers at all).