A lot of your questions are duplicates here so I'll cover them very briefly. Suffice to say that all of this has been a decades-long arms race between exploit developers and OS / compiler developers.
Initially there were no exploit protections. You could find a stack buffer overflow bug, use it to overwrite the current call frame's return address on the stack, wait for the exploited function to reach its end and return, thus gaining control over the instruction pointer (IP), which you could point back at instructions you put in the stack buffer as part of your exploit payload (shellcode), and thus execute arbitrary code.
Then there came stack cookies. These are randomly generated values that are placed just before the return pointer. At the end of each function (in the epilogue) the function would check that the cookie value was correct before returning. If you blindly overwrote a stack buffer to overwrite the return address you'd end up overwriting the stack cookie, which would then be checked at the end of the function and you wouldn't get code execution. To get around this, attackers instead started targeting exception handling structures (SEH) which typically appeared at the end of the stack frame but before the return pointer. By overwriting the SEH structures you could trick the OS into executing an exception handler at an address you controlled, again usually the stack buffer you overwrote with shellcode.
To fix this, OS developers took a couple of approaches. One was SafeSEH, which maintains a read-only table of valid exception handler addresses in a table as part of the executable structure, and verifies that the exception handler address is one of the ones in that table before redirecting control flow to it. On x86-64 Windows systems the stack is no longer used at all for storing SEH structures, which mitigates this attack completely, but I'm getting ahead of myself a bit here.
In the meantime some enterprising exploit developers had started to look into heap (i.e. dynamic memory that isn't the stack) buffer overflows and what could be done with those. Heap buffer overflows sometimes allow you to overwrite pointers (e.g. to event handlers, callbacks, etc.) that led to control over the instruction pointer and thus code execution. Again the attackers would usually put their shellcode in the same buffer they were overflowing.
The other approach I mentioned above was Data Execution Prevention (DEP). This is a bit of a confusing one because in the early days Microsoft called SafeSEH "Software DEP" despite not really being DEP at all. So if you see "software DEP" written somewhere, they mean SafeSEH. Hardware DEP, usually just called DEP these days, is a feature built around a hardware extension that enforces page access flags in the processor. The feature is called NX (No Execute) by most people but it's also sometimes referred to XD (eXecute Disable); I'll just be calling it NX. What NX does is make it impossible for the processor to execute pages that have been marked as non-executable. If you try to redirect the instruction pointer to a non-executable page the processor just throws a protection fault. Operating systems used NX to implement DEP, which marks the stack and data segments as non-executable. This made it impossible to just drop shellcode into a stack or heap buffer and execute it there - those pages were now non-executable and the CPU would refuse to execute the code.
This is where ROP, otherwise known as ret2libc (a Linuxism for the same thing), came in. Instead of trying to put shellcode into the stack or heap and execute that, ROP instead found small chunks of library code that could be chained together in order to get code execution. The initial part of the exploit would gain control of the instruction pointer somehow (e.g. via heap corruption, UAF, etc.) and either fill the stack with the ROP chain data or overwrite the stack pointer so that it pointed at the heap buffer containing the ROP chain data (this is called stack pivoting). Either way, the stack pointer ends up pointing to the top of the ROP chain. Each "gadget" in the ROP chain is a small snippet of code in the format "do something, return". By putting the addresses of these gadgets one after another on the stack, the application jumps to the first gadget, executes a few instructions, hits a return instruction, which then reads the next gadget address off the stack and transfers execution to that, repeating until the end of the ROP chain. Usually the chain would call some function like exec on Linux or CreateProcess on Windows. Another way to go is to call the memory protection function for the OS (e.g. VirtualProtect on Windows) and use it to mark a section of memory that you control as executable, then have the last item in your ROP chain redirect to that memory so you can execute shellcode. Neat, right?
The key part of this attack is that you need to know where these gadgets are in memory. In order to make this difficult (or borderline impossible), OS vendors introduced Address Space Layout Randomisation (ASLR). This randomises the base addresses of various important bits of memory like the stack and executable modules. More and more stuff has been randomised and the randomisation has become less predictable as ASLR implementations have improved. The idea is that if the attacker can't know where modules are in memory for the target process, they can't build a ROP chain. Unfortunately, however, it only takes one non-ASLR module to be loaded in a process in order to break ASLR. In newer operating systems (e.g. fully patched Win10) you can configure a policy that forces ASLR for all modules, but this is not the default (as far as I know). ASLR can also be bypassed using a pointer leak. These occur when an address of some executable memory in a process is leaked to an attacker (e.g. by a UAF bug, or by poor API design). The attacker can then calculate the module base from that leaked pointer and use it to build a ROP chain.
JOP is extremely similar to ROP. It is useful when stack protections are in use, thus preventing stack buffer overwrites, stack pivoting, or return address filtering (a form of partial control flow enforcement). This allows heap-only exploitation via heap corruption, UAF, etc.
Instead of allocating a bunch of ROP gadget addresses on the stack and using a RET of the end of each gadget to jump to the next, JOP uses a jump table in the heap (identical to a ROP chain but with JOP gadgets instead) and a dispatcher. Each JOP gadget is a chunk of library code in the form "do something; indirect jump", e.g. add rcx, 4; jmp [rdi]. Instead of each gadget directly jumping to the next, like in ROP, each gadget instead jumps back to a "dispatcher" gadget.
For example, consider the following two instructions:
dispatch:
add rax, 8
jmp [rax]
This instruction pair (or a compatible one) should be fairly easy to find in library code somewhere. If an attacker can set rax to the address of their dispatch table (e.g. a heap buffer they've filled), and set the instruction pointer to the address of dispatch, then they can chain together JOP gadgets to gain code execution as follows:
- Attacker loads dispatch table into known heap address.
- Attacker exploits bug that sets
rax, rdx, and rip so that rax points to the dispatch table allocated in step 1, rdx points to the dispatch table gadget as described above, and rip is also redirected to the dispatch table gadget.
- The dispatch function increments
rax, so rax now points to the next entry in the dispatch table.
- The dispatch function jumps to the address at
rax, which executes the gadget there.
- The gadget does something and then ends with
jmp [rdx]. This redirects control back to the dispatch gadget.
- Go to step 3, repeat until chain is completed and exploit succeeds.
The registers here (rax and rdx) are just examples, they could be swapped for any general purpose register.
You can read a more full description of JOP in the paper "Jump-Oriented Programming: A New Class of Code-Reuse Attack".
In short, though, JOP is only really useful when the stack cannot be abused due to stack protections being in place.
JOP can be mitigated using control flow enforcement, such as Control Flow Guard (CFG). Protections such as CFG identify indirect call sites in the program and build a table of valid target addresses that they can point to. For example, if you have a callback in C++ defined as typedef void (*SomeCallback)(int, int); the compiler knows that only functions with that signature (e.g. void MyCallbackImplementation(int foo, int bar) { ... }) are valid targets. The compiler then inserts a check before each indirect jump instruction that ensures that the target jump address is valid. If not, it terminates the program.
One additional ROP/JOP trick is to use unaligned gadgets, i.e. gadgets that aren't made up of instructions as they were intended to be interpreted when the compiler generated them, but instead are valid instruction patterns that happen to appear when you jump in the middle of an instruction. These can also be mitigated to some extent by the compiler but I am unsure how commonplace this particular type of protection is.
Hopefully this gets you up to speed. I've glossed over some stuff as this answer is already very long and covers a lot of different anti-exploitation topics very quickly. Some details are missing so I suggest you try to bypass each protection yourself one by one so that you gain an appreciation for all the little nuances.
