Custom socket server on the internet running as root

Question

We are writing a custom socket server which runs on a high port. Until recently, it has been running behind a corporate firewall. Now, it has been decided that the server should be taken outside the firewall, and serve internet traffic. As written today, the server directly mmap's /dev/mem, which requires root.

This feels like a terrible liability.

Without knowing specific exploits, it's hard to make a compelling argument against this decision beyond handwaving with "best practices," "principle of least privilege," "execute arbitrary code as root," or simply "this is f****ing insane."

How can I convince management that we need to run the server as a non-root user, even if it means significant code changes and schedule delay?

Answer 1

That the code "runs as root" is mostly irrelevant. Root or non-root is a distinction that makes sense only locally to a machine, and only if you want to contain some potentially hostile code (e.g. hijacked server code) without bringing down the whole machine. This is the mainframe model from a few decades ago. At that time, it was believed that you could have a Unix (or Unix-like) system such that non-root process could be kept apart from each other, without any possibility to evade that containment and reach other process on the same machine.

This belief hardly holds nowadays. Local privilege escalations are plentiful; it is very hard to plug them all while maintaining a fully functional environment. In general, it is safer to assume that any hostile code running on the machine, under any user, will be able to take control of the whole system, unless some stronger containment measures are applied, such as (in increasing levels of complexity and containment) chroot, jail, virtual machines. In any case, a process that calls mmap() on /dev/mem can gain full control on the local machine (possibly the virtual machine if you go the VM road).

The conclusion of all that is that if your machine is hacked into, and making it Internet-facing sure increases that likelihood, then you should ask yourself what could happen if it gets subverted by hostile outsiders. Changing the code not to run as root and not to mmap /dev/mem does not substantially change things here. What matters is whether the code has been well designed, reviewed, tested, documented and maintained.

I must say that a direct mmap() on /dev/mem sends a strong signal of "these guys should not be allowed to go near a keyboard". I don't know why you are doing that in your server(*) but it seems real weird. Mapping physical RAM in your address space is not a security issue, unless you cling to the outdated mainframe model; however, doing very funky things is a security issue because it makes design, reviewing, testing, documentation and maintenance much harder.

^{(*) My best guess is that your server interacts with a custom piece of hardware that lacks a specific kernel driver, and does I/O by mapping its circuitry in the physical address space, like, say, a graphics card. It would be cleaner, and thus safer, to actually write an appropriate kernel driver.}

Answer 2

Frankly, the relevant words are "custom code running as root and exposed to the Public."

To justify the coding effort and delays, you will have to do some quick calculations on the impact of the code being exploited and a malicious actor gaining root access to the server it is running on. If the cost of a breach is more than the cost of the coding effort and the scheduling delay, then you have a clear metric to bring before the business, and they can make the call on the risks they are willing to take on.

An important factor to consider is the probability of a malicious actor gaining root access directly from your custom code as opposed to gaining access to the non-root user the code could be running as. You also need to factor in the probability of escalating to root from the non-root user. If getting access to the non-root user has the same impact as getting root access, or if escalating to root from the non-root user is trivial, then there is no difference between the 2 users.

Speaking to the business can be very difficult because their perspective is very different from a technical perspective, and they have to balance more issues than just the technology. Finding a common language, like money, tends to help provide a common ground.

Answer 3

You might try demonstrating server problems that occurred in well-tested servers and make the claim "If after all of the testing and examination they were insecure, how can we expect to do better?"

The first example that pops into my head is the Shellshock bug. When combined with CGI from the Apache Web server, it allowed remote execution. This was not due to a bug in Apache, but still Apache became vulnerable. If you followed best practices and ran Apache as an unprivileged user, this was only a bad vulnerability. If you ran Apache as root, it was a fatal vulnerability that seriously jeopardized the safety of your server and whatever was connected to it.

PS: I agree that saying "this is f****ing insane." should be more than enough :)

Answer 4

It is really important that you provide viable alternative solutions and not just rely on arguments that it is a bad technical solution. For example, could using a VPN setup be a better choice rather than re-developing the code to use a non-privileged user? Can you generalise the solution? For example, would a VPN solution provide increased and more secure functionality in other areas as well. Don't just argue that it is a bad idea, argue for a better idea which gives them a solution they want. To do this, you will need to understand what it is they actually want - not what they have asked for, but what the underlying driver is. Often you will find that what they are asking for is the wrong thing, but the functionality they want is necessary. Understanding this will allow you to identify a better solution which they will be happy with.

The other thing you need to do is prioritise the business costs/risks over technical ones. You can include the technical based risks/costs as well, but don't focus on them. As soon as you start talking privilege escalation, CVEs, data encryption, firewalls, ports etc, you will lose the interest of the business managers. Instead, talk about things like lost sales/productivity, reputation damage, theft of trade secrets or proprietary information, reduced customer confidence, government and other compliance etc. Essentially, put your case in terms which management can understand - business terms.

If all else fails, consider arguing for a penetration test or external audit. The unfortunate reality is that all too often, management will not listen to in-house technical recommendations, but will sit up and take notice as soon as a reputable external company or consultant points out the risks. This is sometimes due to the external company/consultant having better skills at presenting arguments in business terms and sometimes it is because they are more willing to present the facts in a blunt and clear way which internal staff may be reluctant to do because it might be interpreted incorrectly or reflect poorly on them. Often this process is extremely frustrating as the external company/consultant will say exactly the same things you have been saying. Don't worry about it - focus on the end goal, which is getting a better solution which you are able to manage and doesn't keep you awake at night worrying about when the brown stuff will hit the fan.

At the end of the day, all you can do is identify and report the risks and ensure they are communicated as clearly as possible to those who make the final decisions. They are getting paid the big bucks to take on the responsibility for those decisions and it is actually quite legitimate for them to decide to accept the risks and go forward anyway. You may still think it is a bad idea, but that is largely irrelevant. For all you know, there might be other strategies or reasons which change the risk equation and make the choice to continue with what you feel is a bad solution a viable choice after all. Provided you have done what you could to communicate the risks and provide viable alternatives, you have done what is required and now need to implement the final decision in the best way you can with the resources provided.