In a Unix-like system, the concept of privileged and non-privileged users is used for security, preventing numerous attacks. When a non-privileged user executes malicious code at a normal level, the kernel can reject its execution by recognizing the user's lack of privilege.

However, it appears to me that this security mechanism is effective only if the malicious code is crafted to exclusively utilize the kernel API. It is crucial to remember that the kernel itself is essentially a sequence of instructions for the micro-controller. In theory, there could be instruction code that bypasses the privilege check of the kernel by executing without utilizing the kernel API. To make it worse, I can't imagine how the kernel could preemptively identify, before execution, whether a particular code segment ultimately involves the use of the kernel non-API.

In theory, the existence of such code seems plausible to me. Is my understanding correct? If so, I am interested in discovering examples of such code or exploring discussions related to historical instances.

  • 2
    To give the essence: kernel and user space are separated at the hardware level (CPU feature) and syscalls are the only way to switch from user to kernel space (i.e. these are special CPU instructions). Because of this separation user code cannot simply call into somewhere in the kernel. For a deeper understanding you might start with Wikipedia: system call and then follow are the various links there which go deeper into the details of the CPU which allow this kind of separation. Nov 18 at 16:32

2 Answers 2


Someone will hopefully come along with a more complete answer, but this will get you started. I would write more but this is typically a 200 level course in CS, and I’m on mobile. I also may do a poor job of explaining. Here it goes…

What you’re missing is that Unix-like operating systems (and most/all other general purpose operating systems) make use of different CPU contexts, which have different privilege levels; these are effectively security controls and they’re enforced by the CPU

Note, when I say “context”, I mean both the general register state AND the control registers which dictate privilege. Not only a simple “context switch”, if you know the term

Yes, you’re correct that the instructions run on the same CPU, but there are instructions that can only execute in the kernel context (and registers only addressable in that context, as well)

These instructions will fault when executed in userland, as userland is not privileged

It’s no coincidence that these privileged instructions (and registers) are critical to performing I/O as well as setting up userland memory mappings, which act as security bondareis (no mapping means no access)

Things you need to learn about to understand this:

  • Protection rings, specifically ring-3 (userland context) and ring-0 (kernel context) in the case of Unix-like operating systems as well as Windows
  • The MMU which is critical for restricting userland access to privileged memory
  • Virtual Memory, closely related to the MMU
  • Interrupts, which can be used as a signal that a userland process wants to perform a privileged operation. They trigger a context switch into interrupt handlers, implemented by the kernel
  • System Calls, the interface for userland to request specific operations implementes in the kernel. There are hundreds of system calls (maybe well over a thousand at this point?) implemented in Linux. Examples of system calls include open(), read(), write(), etc. Invoking a system call from userland typically involves setting a specific general purpose register to a system call number and then invoking a specific interrupt number (via a break or trap instruction, depending on the CPU architecture)
  • Those are good pointers, and I like what you pointed out that I missed. Below I tried to make sense of the whole process; is it correct?: The bootloader, loaded at boot, gains CPU privilege. It then loads the kernel and instructs the CPU to grant privilege to the kernel exclusively. Then the CPU executes privileged instructions solely for the kernel. If a user attempts to bypass and execute code, it must pass through the kernel (otherwise CPU denies); during this process, the execution is "tagged" with a user context, allowing the CPU to discern and restrict certain non-API instructions.
    – Student
    Nov 18 at 22:36
  • You probably understand the principles, but to properly articulate it, think of it in terms of continuous transitions/context switches. You would benefit from reading en.m.wikipedia.org/wiki/Protected_mode or put another way, replace the bit about a process being “tagged” with “executing without the mode bit set”
    – adam
    Nov 24 at 17:08

In linux, a process runs in a specific context. This context includes memory assigned to the process by the kernel and a permission set including user ID.

When the process executes, it only has access to the memory assigned to it and to kernel system calls. There is no way (in software at least) to "bypass privilege check". What is it going to do? Access to devices requires a kernel system call. Access other processes' memory will either be blocked by the cpu page table permissions or it requires a kernel system call to request access to the memory. Privileged instructions reserved for the kernel are also blocked by the cpu.

There are some hardware ways to bypass the kernel (search for "row hammer"), but this is probably out of the scope you were thinking.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .