Old OS memory space protection - was it really that bad?

Question

In his book Security Engineering, Anderson really focuses on how in the 90s and early 2000s programs would need to access memory that wasn't their own, and programmers programmed with the assumption the program would be run with administrative privileges:

The latest Microsoft system, Vista, is trying to move away from running all code with administrator privilege, and these three modern techniques are each, independently, trying to achieve the same thing — to get us back where we'd be if all applications had to run with user privileges rather than as the administrator. That is more or less where computing was in the 1970s, when people ran their code as unprivileged processes on time-shared minicomputers and mainframes. Only time will tell whether we can recapture the lost Eden of order and control ... , and to escape the messy reality of today... but certainly the attempt is worth making.

In particular this quote

PCs carry an unfortunate legacy, in that the old single-user operating systems such as DOS and Win95/98 let any process modify any data; as a result many applications won’t run unless they are run with administrator privileges and thus given access to the whole machine

Does this mean that ideas such as these, where you can assign any address to a pointer and have access to the memory would work on Windows 95/98? Why would they design it this way as what would be the benefit of reading some other programs memory (or writing to it!)?

int * firefoxmemory = (char*) 0x11111111 //this is just an example of address.
*firefoxmemory = 200;//screw firefox

Answer 1

Memory isolation

Your example wouldn't work on Windows 95, but it did work on DOS and Windows up to 3.11 (not Windows NT).

The PC architecture, and the Microsoft series of operating systems, started with the Intel 8086 processor and an operating system (DOS) designed to run a single program at a time. You would run a program, and when you were finished with it, you'd exit it, so the overwriting data of another process wasn't really a concern. (You could have programs that remained in memory after exiting; they were essentially part of the operating system.)

The Intel 80386 processor changed the deal by being the first of its lineage to have a memory management unit (MMU). An MMU is a hardware component that provides virtual memory by translating virtual addresses (pointers in a program) to physical addresses (actual locations in RAM¹). Windows 95 was the first in its lineage to take advantage of it: each Windows 95 application ran with its own MMU configuration, so that *firefoxmemory would either point into the program's own address space (in which case whatever bad thing it did could only affect that program) or not point anywhere at all (in which case the program — and not the whole system — would crash).

Windows 3.0–3.11 took advantage of some of the 386's novelties, but they ran all applications in the same address space. This was due to several factors:

• a requirement to keep old Windows 1.x/2.x applications running (they were designed to run on single-address-space hardware, and many took advantage of that to poke into OS internals to achieve things that weren't possible through documented interfaces);
• lack of development time to redesign the whole operating system based on a completely different architecture;
• a requirement to run on computers that had little RAM: keeping applications isolated does cost more memory on a scale of 2 MB or 4 MB (because data has to be copied between programs, and because the operating system has to keep track of all program interactions to allow applications to communicate and to regulate the communication).

It wasn't so much that the operating system was explicitly designed to allow programs to access each other's memory, but that there was no way to prevent it. When ways to prevent it became affordable, it took a few OS versions to take advantage of them.

For example, let's consider a feature such as inter-program copy-paste (clipboard). If you don't have memory isolation, this can be implemented by having the source program keep a copy of the data in its own memory; when the data is pasted in another program, that other program copies the data directly from the source program. The only thing the operating system needs to track is who currently owns the clipboard. If applications' memory is isolated, the operating system needs to arrange for the data copying, and possibly to store a copy of the data independently of the source application. This requires more development work to write this code, and more resources to store this code in memory and run it.

I've simplified a number of things here; for more about how an operating system uses an MMU to isolate applications, you can read How the kernel can prevent a malicious program to operate?, How can two identical virtual addresses point to different physical addresses?, Is it possible to support multiple processes without support for Virtual memory? (they're Unix-oriented but the principles apply to Windows or any OS you're likely to encounter on a PC).

Privilege isolation

In this passage, Anderson isn't actually discussion memory isolation, but privilege isolation, where a running program is prevented from affecting certain parts of the system. Memory isolation is necessary, but not sufficient. The operating system must also control the ways processes interact, control what files they open, etc. Even with each application running in its own memory space, applications can interact. Memory isolation merely forces applications to use operating system services (via system calls to a kernel which can access all memory) to interact.

The Windows 1.x/3.x/9x/ME series of operating system was designed as a single-user operating system and did not isolate applications. Memory isolation was added in Windows 95, but only to improve stability, not to implement security restrictions. These operating systems have no security restrictions: if you ran a program on your machine, it is allowed to do everything. (This is of course helpful to virus writers.) The memory space is isolated, but not the file space.

The Windows NT/2000/XP/… series of operating systems, like operating systems in the Unix family, was designed as a multi-user operating system, which introduces a basic security goal: what a user does should not adversely affect what other users can do. So the operating system has to enforce, for example, that programs executed by Alice cannot interact with programs executed by Bob, cannot modify Bob's files, etc. Programs and data that are part of the operating system concern every user and so should be protected from all users. (Of course, some interactions are allowed by explicit permission, e.g. a network server does process requests that it receives from the network, Bob can change a file's permissions to be writable by others, some users are granted administrative privileges and so can modify the operating system, etc.)

A great many applications were written to target Windows 9x, which had no security restrictions. So they took liberties such as writing to the operating system directory (under Windows 95, copying the file to the Windows directory was the normal way to install a shared library). Much of what these applications do was considered somewhat messy even by the standards of that time, but it worked so people did it. Versions of Windows from the NT series (which includes all versions since XP) necessarily broke some of these applications, though Microsoft did add a number of workarounds (such as pretending that writing a file to a shared directory succeeded, but actually writing it to a user-specific directory).

Windows XP enforced user isolation, but most installations wouldn't take advantage of it: many installations just had the user be an administrator, so all the programs they ran had the privileges to do pretty much anything. There were two main reasons for that:

• It allows badly-behaved legacy applications to run. While most such applications are only used by a small proportion of users, many users do run such applications and it takes a long time and a huge effort to renew them all.
• Introducing privileges means that sometimes the user will be told that something can't be done because of a lack of privileges. Witness the complaints about Vista, that it would keep showing those privilege escalation prompts.

Application isolation

If you're young enough to have discovered computers through portable devices, you may be used to a model of isolation centered on applications, rather than users. In the Unix and Windows model, Alice's data is protected from Bob. Applications are neutral in the security model: they just execute under the identity of the logged-in user. This means that the application code has to be trusted not to misbehave.

Windows has traditionally coped with misbehaving applications by trying to detect and contain them through antivirus software. That doesn't work too well. Unix systems, especially Linux, tend to cope by distributing more software through traceable channels from which malware is eliminated.

Operating systems designed for mobile devices, in particular iOS and Android, don't trust application developers, so applications are isolated; for example each application has its own file space. This has a security benefit, but also a significant cost in that it reduces what applications can do. Applications require privileges to modify operating system behavior (e.g. to create shortcuts and more generally automation), you can't easily manipulate the same file in different applications, you can't easily allow multiple applications to display information at the same time, etc. Or, to take a more dramatic example, you can't write a debugger with restricted privileges, because the whole point of a debugger is to snoop on and perturb the execution of the application that's being debugged. This cost is why the mobile application isolation model can't just be grafted onto a PC environment.

Answer 2

If one goes back a few years before Windows, it was pretty much expected that any program running on a microcomputer would "own" the computer. If it wanted certain services to remain usable it would have to leave certain parts of the system alone, but otherwise there wasn't really any need to "protect" anything. Mainframe computers were sufficiently big and expensive that letting one user monopolize the machine would be impractical, and when multiple people were using a machine simultaneously it was necessary that it be able to act like several independent computers which wouldn't disrupt each others' operations, but on a microcomputer that wasn't an issue. Unlike today's computers which all have hard drives or non-volatile storage, early microcomputers had neither. If you powered a machine down and back up, its memory state would be effectively reset to factory default. Power on an Altair 680 and it will wait to receive a program from the serial port in S-record format. Feed it such a program (most likely using a paper tape reader) and and it will run it. If something goes wrong, power off the machine and reload it from the paper tape.

When computers added floppy drives, the same principles continued to apply. It was only after personal computers got hard drives that issues of security started to become relevant, but many of the programming techniques which made sense with floppies continued to be used because they worked.

Answer 3

A simple example: The DOS screen-writing capabilities left a lot to be desired. It was routine to simply write video memory directly. You needed to do this to get reasonable performance for anything that does a lot of screen updates and it was absolutely essential if you wanted to write the bottom right cell of the video screen--say, to draw a border around what you were displaying.

In the earlier versions it was also necessary to write to the memory of the control block just in front of your program if you wanted to open a lot (over 16??) of files. I forget what version added an OS call to accomplish the same thing.

Answer 4

To expand on the MMU answer, all of the early 80s "personal computer" chipsets started off without that functionality. It has a non-zero cost in chip area and delay to memory, a cost that was a lot higher in those days on the 3 µm chip process. Remember that we can fit about 20,000 transistors today in the space of one back then.

The 8086 and 68000 (as used in the Mac and Amiga) did not have MMUs. They also only had a maximum of one megabyte of memory and usually only a floppy disk drive (so no disk-based virtual memory either). With the lack of networking and only a single user, what would be the point of memory protection? You'd only be protecting the user from themselves, and preventing some potentially useful inter-app data sharing. The common mode of usage was to load a program from disk and then hand the entire machine over to that program. DOS didn't even have multitasking; there was a "terminate and stay resident" mode which allowed programs to leave code behind to run on a keypress or timer.

MMUs are still not present in most microcontrollers, and are an optional feature on ARM.

Video memory was a common target for direct access; if you wanted to write a game, it was considered the only way to achieve acceptable performance. You also had to directly write to sound hardware (if present) as there were no drivers.

There was also the strange intermediate era of "DOS/4GW" before Windows acquired the DirectX subsystem. In order to overcome the hardware limit of 1Mb of memory imposed on 16-bit programs, you could ship the game with a small 32-bit operating system of its own. This required you to quit Windows 3.0.

I still have my 16-bit games code from this era. I keep meaning to post it to github.