49

Edit: though my wording may not be exact, I understand that two containers don't have access to the same memory at the same time. My question is can one container see data that another has deallocated?

LXC allows us to over-provision RAM.

I assume this means that if one container needs a bunch of RAM and another container isn't using its allotment, then the unallocated RAM goes to the container that needs it.

Let's say that one container had some private keys loaded, and that memory was deallocated, and another container just allocates its maximum heap and starts walking it.

Is there the possibility of reading that private key?

Or is it wiped or otherwise allocated in a way that prevents data leakage?

Where is the documentation that clarifies this?
(my serach-fu is weak on this - probably because I don't know the right terms)

8
  • 17
    For your mental model, it's also important to note that containers are not virtual machines. Containers do not manage their own pool of memory. Containers are just a group of one or more processes, with some extra Linux security features enabled. For example, containers will typically use "cgroups" to limit their resource usage. Just like normal processes cannot access other processes' address space, containers (which are processes) cannot do that, and are constrained by the kernel's security model.
    – amon
    Apr 4, 2023 at 9:07
  • 2
    LXC containers contain processes, and memory given to processes is cleared beforehand, so the answer is "no" - LXC doesn't weaken the existing process isolation model. Apr 4, 2023 at 15:49
  • Is the hypothetical container here started with the privileged flag? (UID 0 doesn't provide that much by way of extra privileges anymore; instead, that's managed by capabilities; Docker drops dangerous capabilities like CAP_SYS_ADMIN, unless you use --privileged). Apr 5, 2023 at 16:33
  • 1
    ...similarly, a lot of ability to read raw memory accrues to any process with the CAP_SYS_RAWIO capability, but that's another thing that's not going to be available in Docker unless you started a container as --privileged. Apr 5, 2023 at 16:36
  • 1
    @amon totally agree but it's worth noting that LXC does now support virtual machines as well as containers ubuntu.com/blog/lxd-virtual-machines-an-overview though this is almost certainly not what the OP was referring to.
    – Rodney
    Apr 6, 2023 at 21:23

3 Answers 3

20

Processes in LXC containers are normal processes as far as the Linux kernel is concerned. They are separated from most of the host's resources by using namespaces, which does not make them a special kind of process. This is different from how virtual machines work.

When Linux (or another OS) allocates new pages to a process, they are zero-filled.

There is a Linux kernel configuration option that allows a process to ask for non-zero-filled pages, which may contain leftover data from other processes, using the MAP_UNINITIALIZED flag. However, this is very rarely enabled - only for embedded systems where all processes are trustworthy and you need every drop of performance.

11
  • Am I correct that, when using namespaces, the root user of the container is mapped in such a way that it does NOT have permission to access the memory outside of its container?
    – coolaj86
    Apr 5, 2023 at 0:19
  • Containers have nothing to do with memory access.
    – user10489
    Apr 5, 2023 at 0:20
  • 2
    You are conflating processes (which cgroups and therefore containers constrain) with memory. Containers have nothing to do with limiting memory access, they only constrain access to processes and namespaces and limit total memory use, not access. As has been said repeatedly in every answer here, processes constrain memory access, not containers. Memory is assigned to processes.
    – user10489
    Apr 5, 2023 at 5:01
  • 1
    @coolaj86 The "root user in a container" is not the operating system's root user, so it doesn't have any elevated permissions. Not an expert on lxc, but I'd assume the user in a lxc container is a namespaced user (a new user in a new namespace) that can't see or do anything outside of its own namespace. Apr 5, 2023 at 9:53
  • 1
    @coolaj86 it is actually namespaces that are relevant, not cgroups. cgroups put limits on max CPU, max memory, max disk bandwidth, etc. Namespaces are what control permissions.
    – user253751
    Apr 5, 2023 at 13:22
82

This isn't how memory allocation in linux works, so your scenario is not right.

The linux kernel maintains a pool of free pages and quickly freeable pages (which includes cached disk blocks and process pages already written to swap but still in ram). Of these pages, it also keeps a pool of pre-zeroed pages (the size of this pool is adjustable). When a process needs a new memory page, it is pulled from this pool.

When a process asks for a new page, it will get a zeroed page, and won't get a page with data from another process. Generally, a process can't access another process's memory, but there are many exceptions (see a partial list below).

While new pages will arrive from the kernel zeroed, some memory management libraries may reuse memory rather than releasing freed pages to the OS, and these might not be zeroed (depending on the library and API call used), containing old data from the heap and stack from its previous use within the process. This sometimes can be a security risk and source of bugs from reading uninitialized variables, but zeroing them is also considered a performance issue, especially if the code using the memory will immediately initialize it anyway.

Overprovisioning doesn't mean ram is given to both programs. It means it is given to neither of them until the last second.

So, let's say program A and B both have 1G of memory allocated to them. The system has 8G. Now both A and B (simultaneously or sequentially) ask for 5G. With overprovisioning, we can grant that request, but not actually give either one the memory -- just the address space.

The system still has 6G of memory in the free pool. A and B have "requested" a total of 6G each, but are each only using 1G and only have that 1G assigned to them. But they each have page table entries (with no assigned pages) for another 5G.

So the first time one of them writes to one of the newly requested pages, it gets a soft page fault, which causes a (pre-zeroed) page to be pulled from the free pool and assigned to that page table entry, and then the soft faulted write is allowed to continue with a real memory page assigned to it.

If the two programs allocate and use memory slowly, and perhaps never use everything they requested, this all works fine. If you have some swap, possibly some unused or infrequently pages in the program get written to virtual memory (swap) and returned to the free pool.

However, if both of them end up with a combined working set of used pages greater than the system physical memory, then either one or both of them will get killed with an OOM error (out of memory) (if you don't have enough swap to cover it), or the system will start thrashing as it tries to constantly swap pages between physical ram and virtual memory.

The alternative to overprovisioning is to immediately deny the memory request if there isn't enough virtual memory to cover it. Many programs are not written to handle this denial and will crash due to bugs, or just crash because they can't continue without the memory. So frequently, at worst, overprovisioning delays the program's death (or makes the system thrash), and at best, it avoids some possible nasty bugs and allows programs that request memory they might not actually use to continue running as if they got it anyway.

Adding containers to this does not change it at all. When you provision the container, you don't assign memory to the container (loading the container and running it does that live, as needed), it assigns memory limits to the container. When enough actual pages are assigned to the things in the container to exhaust those assigned limits, then the things in the container will get an OOM kill just like above.

If you've overprovisioned the containers and they try to reach those limits all at once, you'll get either thrashing or the OOM kill when the system's memory is exhausted, before the container's memory limit is reached.

It is also possible to tweak the container memory allocations so that one container can thrash while other containers perform normally.

Here is a partial list of cases where a process can see another process's memory:

  • Immediately after a fork, the parent and child share all memory pages. The linux kernel marks these as copy on write (with a reference counter) so these pages are shared read only, and the first write by either process clones the page so it is no longer shared.
  • A process can clone itself, sort of like fork, but with more control over which parts of the process are shared writable and which parts are COW cloned (as with fork). If almost nothing is cloned, it acts like a thread or light weight process.
  • A process can explicitly share a page to another process through multiple mechanisms, and this can have full bidirectional write for both processes. (The oldest form of this is SysV shared memory which is all but obsoleted by other more flexible methods.)
  • A process can debug (ptrace) another process and get full access to its memory and execution flow. However, since this is such a huge security risk, this is generally only allowed for root or for for a parent process to debug its child; The main use is for a debugger (like gdb) to start a process to debug. However, programs like strace and ltrace can do this without root access. And this can be relaxed via kernel option so gdb can attach to any running process a user owns.
  • A program can transfer a page of memory to another process via pipe or socket, but this acts more like a copy than a real sharing, especially if the receiving program doesn't try to read it with the same page/byte alignment.
  • The shared library system is entirely based on multiple processes sharing read only pages of libraries, and obviously, the executables of processes are also shared this way.
12
  • 3
    I appreciate the thoroughness of the answer, and there is some nuance there that I wasn't aware of before, but my question is whether or not it's possible for a container to allocate memory that has data in it from prior use by another container (or the host), or if it is sanitized by things like the kernel user namespace protections.
    – coolaj86
    Apr 4, 2023 at 1:10
  • 16
    Answered that, guess its too buried. New pages come from the pool of free pages already zeroed. The kernel doesn't hand out pages before they are blanked, and overprovisioning doesn't cause pages to be shared. Having said that, there have been bugs that caused unzeroed memory to be leaked, but once discovered, those bugs are squashed quickly.
    – user10489
    Apr 4, 2023 at 1:17
  • I've proposed an edit adding a section to your answer to summarize what I now believe from a more careful reading of your response. Would you please take a look at that and correct any poorly worded or incorrect language?
    – coolaj86
    Apr 4, 2023 at 1:40
  • 2
    If the namespace is working correctly and there are no leaks, processes in the container can't even see processes outside the container, let alone access them.
    – user10489
    Apr 5, 2023 at 5:08
  • 1
    Linux kernel uses lazy page allocation mechanism and it gives zero-filled pages to the userspace. Just for the reference: github.com/torvalds/linux/blob/… Our point of interest is the function static vm_fault_t do_anonymous_page(struct vm_fault *vmf) Apr 5, 2023 at 13:42
4

Yes, if memory used by a process in one LXC container is later allocated to another process in another LXC container, the contents will definitely be wiped. This is the case for all processes and has nothing to do with containers.

You asked: "Where is the documentation which clarifies this?" I doubt that there is any specific documentation which addresses your question; if one understands how 'containers' are implemented, then it is obvious.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .