What can I use to avoid running a full-fledged VM that gives me comparable security?
Possibly, something like https://coreos.com/rkt/ but I don't know what their security properties are.
Classic VM: There are two kernels, the hypervisor uses CPU feature to emulate a hardware, and the guest kernel is generally unaware that it is running on a virtualized system. This is generally considered to give the best isolation, however giving restricted access to the guest system can be tricky as many security features needed to be enforced by the guest system.
Paravirtualization: There are two kernels, the guest kernel is aware that it is being virtualized and it uses a paravirtualization driver that is specifically designed to talk to the host kernel/hypervisor for hardware access instead of trying to access an actual hardware. This is middle of the road in terms of isolation and flexibility.
Container: There's only one kernel, the kernel creates a namespace for each container so that the containers can't see each other. This gives you the least isolation, as any security bugs in the kernel can break out to the host system, but also most flexible if you need to define complicated access restriction. For example, with container you can define an SELinux policy that will be applied to the guest system but enforced by the host system.
If you want to run a known malware, I suggest using a full blown virtual machine. As currently implemented, container technology like rkt and docker isn't intended to be unbreakable by privileged processes running inside the container. However, note also that there are a certain class of malware called blue pill malware that are designed to break out of classic virtualization by exploiting flaws in the hypervisor. While these are quite rare, it makes it not advisable to run malware samples if you have zero clue what it is supposed to do.
It rather depends what you mean by "comparable security".
One key difference between a VM and other approaches is that the former simplifies addressing availability; it's possible to migrate a running VM across different hardware nodes, it's easy to snapshot the state of a running VM to a disk image.
Ignoring these for now (it possibly makes this too complicated a question to answer here):
1) Use the OS properly. Unix has always been a multi-user/multi-tenant operating system. It has a sophisticated and mature model for users, files and processes. You can control the available memory, scheduling, storage and CPU usage. You can also have firewalls configured for specific environments on the host (via netgroups).
2) run the required software in a chroot environment (optionally with elements from above)
3) run the required software within an apparmor environment (optionally with elements from above)
4) run the required software in a container (optionally with elements from above)