A long time ago people talked about the Harvard architecture, which had the program separate from the data. Similarly, if the software of a server was not modifiable (burned in to a crystal or something - note: hypothetical, do not attack this premise) and it came out of the box with no root user and no way to add or change anything (except data), we would have a sort of "appliance" that worked like any other piece of familiar hardware, and could not be commandeered.
Does anyone do this now and if not, is there a technical reason why not? Basically I am proposing a throwaway device that perhaps does not even store the data in itself, but only controls access to a separate disk farm or something. If something goes wrong, you unplug it and plug a new one in and keep going, like RAID for the server itself.