GreenHills Software is a company that specializes on creating OSes for military embedded systems. Their INTEGRITY system uses a microkernel (less code with less bugs) that has been formally verified (even less bugs).

Such special-purpose domains, where only the most basic software components can be re-used, are probably the only area where it is manageable to deploy a custom OS. Virtually everywhere else, the requirements for interfacing with the outside world, with regular users and with new hardware every few years will completely eat up all your development resources.

Moreover, to my knowledge it is still not well understood how to manage very large software projects. If you have the money and the people to build a new OS in a few years, chances are it will be much more expensive, take much longer, and completely borked.

Finally, there are not a lot of things where a new custom development could improve. After accommodating all the user's and policy enforcer's requirements, you will probably notice that your specialized OS becomes pretty darn similar to existing solutions. For example, its cool to imagine your minimal OS with a few thousand lines of code. Lets say you have that secured. Now you want a web browser with SSL/X509. The most simple implementation of those will likely be multiple times the size of your kernel. And if you don't use existing libraries it will be riddled with compatibility and security bugs.

The main area where OS design could be improved today is - maybe - to move towards a microkernel OS. A design as it was proposed in the Perseus and Nizza architecture could allow you to run security-critical applications with high isolation assurance next to your regular commodity OS, and let the commodity applications refer to the secure applications for tasks like signing documents, establishing session keys etc. More recent examples of this are Genode, TrustVisor and Qubes OS. However, to make such design scalable to many applications in many highly isolated compartments, you need a modern microkernel. So Qubes and TrustVisor are already out the door.

GreenHills Software is a company that specializes on creating OSes for military embedded systems. Their INTEGRITY system uses a microkernel (less code with less bugs) that has been formally verified (even less bugs).

Such special-purpose domains, where only the most basic software components can be re-used, are probably the only area where it is manageable to deploy a custom OS. Virtually everywhere else, the requirements for interfacing with the outside world, with regular users and with new hardware every few years will completely eat up all your development resources.

Moreover, to my knowledge it is still not well understood how to manage very large software projects. If you have the money and the people to build a new OS in a few years, chances are it will be much more expensive, take much longer, and completely borked.

Finally, there are not a lot of things where a new custom development could improve. After accommodating all the user's and policy enforcer's requirements, you will probably notice that your specialized OS becomes pretty darn similar to existing solutions. For example, its cool to imagine your minimal OS with a few thousand lines of code. Lets say you have that secured. Now you want a web browser with SSL/X509. The most simple implementation of those will likely be multiple times the size of your kernel. And if you don't use existing libraries it will be riddled with compatibility and security bugs.

The main area where OS design could be improved today is - maybe - to move towards a microkernel OS. A design as it was proposed in the Perseus and Nizza architecture could allow you to run security-critical applications with high isolation assurance next to your regular commodity OS, and let the commodity applications refer to the secure applications for tasks like signing documents, establishing session keys etc. More recent examples of this are Genode, TrustVisor and Qubes OS. However, to make such design scalable to many applications in many highly isolated compartments, you need a modern microkernel. So Qubes and TrustVisor are already out the door.

Addon: Just noticed that everyone is focusing on US and making kernels, so maybe some side note: The US, the German, the French and probably also most other governments have been and still are looking into hardened OS for certain purposes. The German government is using the SINA box, which is a hardened Linux plus smartcard that implements a VPN gateway for not-quite-so-critical tasks. Makes you wonder what they use for critical VPNs. They have funded research in alternate OS design. Today they are funding virtualization for Android, so you can run a isolated compartment on an otherwise mostly standard Android phone. SELinux by the US is well-known and the french government has a similar system. The NSA is now also trying to get SELinux into Android. The fact that several millions are spend this way bascially confirms aforementioned problems.