I know what is real and protected mode of cpu execution. But how CPU implements protected mode ? AFAIK cpu does everything what the program says it to do. Where is the protected mode behaviour implemented ? Does OS plays any role in this ?
There's a bit in a register that indicates whether the CPU is currently in kernel mode or user mode. Program code cannot modify this bit directly: it has to use special instructions that do more than just change this bit. Certain instructions can only be used in kernel mode; attempting to use them in user mode instead triggers a trap.
While the processor is running in kernel mode, it can switch to user mode at any time. But while the processor is running in user mode, there are only a few ways to switch to kernel mode: with the system call instruction, because of an interrupt, or through a trap. In all cases, the processor doesn't just switch to kernel mode, it also jumps to an address in memory which can only be configured by kernel code. This ensures that user mode code cannot execute whatever it wants in kernel mode. User mode can only call a few specific entry points in the kernel code, and the kernel code at these entry points can set up its processing environment carefully and validate its inputs.
All of this would be useless if user code could modify the content of memory that contains kernel code or data. It cannot due to the memory management unit (MMU). One of the functions of the MMU is to limit what memory is accessible at a given time. Memory can be marked as accessible to the kernel only, and switching between kernel mode and user mode may change the access permission table. The MMU configuration can only be modified while running in kernel mode. This way the kernel can configure the memory so that it can access its own code and data, but user code cannot.
The kernel actually maintains one MMU configuration per user process. Before switching to user mode, the kernel modifies the MMU configuration to correspond to the one for the process that will be executing after switching to user mode.
Pretty much any sufficiently advanced processor follows this basic model: not only any PC manufactured in the past 30 years, but also any smartphone and even many embedded devices.
I've simplified a lot of things. In the real world, there can be more than two privilege levels, more processor modes, more aspects of MMU configuration, etc. You have the gist of how it works. If you want to understand kernel code or write your own, you'll need to read about a specific processor architecture in more detail.
“Real mode” is a historic aspect of x86 processors. It hasn't been relevant in the real world for the past couple of decades except in some rare embedded x86 systems and to some people writing bootloaders or running some really old legacy software.
Protected mode and real mode are two submodes of the Legacy Mode of the x86-64 architecture. One of the differences between them is in how physical addresses are calculated. Another important difference is that protected mode introduces the concept of privilege levels and read/write/execute protection attributes to isolate and protect different programs from each other. The CPU uses the value of CR0[PE] (CR0 is just a register) on each memory access (memory data reads, memory data writes, and code fetches) to determine whether it is in real or protected mode. If CR0[PE] is 0, the CPU is in real mode. Otherwise, if CR0[PE] is 1, the CPU is in protected mode. The OS can change the mode using the
MOV CRO instruction. All instructions that follow
MOV CRO will be executed in the new mode. For more information, you can refer to the Intel SDM Volume 3.