Recently, I have heard of a new virtualization tech called containers. Suppose the container has been compromised, does this mean the host is also compromised (since the container is a process on a host)? In terms of security, is a VM (virtual machine) more secure than containers?
If the kernel is compromised in the container, the host is compromised.
Ostensibly, a compromised container should not be able to harm the host. However, container security is not great, and there are usually many vulnerabilities that allow a privileged container user to compromise the host. In this way, containers are often less secure than full virtual machines. That does not mean that virtual machines can't be hacked. They are just not quite as bad.
If the kernel is exploited in a virtual machine, the attacker still needs to find a bug in the hypervisor. If the kernel is exploited in a container, the entire system is compromised, including the host. This means that kernel security bugs, as a class, are far more severe when containers are used.
Containers are often implemented by using namespaces:
A namespace wraps a global system resource in an abstraction that makes it appear to the process within the namespace that they have their own isolated instance of a global resource. Changes to the global resource are visible to other processes that are members of the namespace, but are invisible to other processes.
Unfortunately, Linux namespaces typically expose a much greater attack surface area from the kernel. Many kernel vulnerabilities are exploitable in namespaces. While not every container solution uses Linux namespaces, they all use the same kind of technology, with the same security issues. Some containers, like Docker, are capable of making use of a syscall filtering framework called seccomp to reduce the attack surface area of the kernel. This is an improvement, but isn't enough.
From Daniel Shapira:
[...] kernel exploits can be devastating for containerized environments. This is because containers share the same kernel as the host, thus trusting the built-in protection mechanisms alone isn’t sufficient.
Because containers are not as isolated as VMs, yes, in one way they are less secure. See forest's answer.
Having said that, I think it's worth noting that containers also provide some some benefits from a application security perspective. Because they typically run a single process they limit the attack surface, there's no cron monitor or ssh daemon running inside the container, for example. Container images are immutable: on restart they load from the original binaries. You can't overwrite the application. Most container technologies also allow precise control over what ports are open and to whom. You can have a container for a DB that cannot be accessed from anything other than your other containers.
All of this assumes that these features work as advertised and do not have flaws, of course. And if an attacker manages to escape from the container, none of the above applies. It adds a bit of overhead but a belt and suspenders approach is possible: containers on top of VMs.