There is always something in memory. Each bit contains either a 0 or a 1 at all times. If the same space in memory is used for two different purposes, the computer (the compiler or the runtime system) can't tell: purposes are a human concept, not a machine concept.
On most systems that you'd think of as computers nowadays (PCs, servers, mobile phones, routers, etc., but not the pervasive kinds of microprocessors and microcontrollers found in most electronic components), there is a memory management unit (MMU). The MMU translates between memory addresses used by the program (virtual addresses) and memory addresses in the actual memory bank (physical addresses). At this translation layer, it's possible to mark some zones of virtual memory as inaccessible. If a program tries to use inaccessible memory, that program will be killed (Windows calls this a general protection fault, Unix calls this a segmentation fault).
It's possible to use the MMU for memory protection. In fact, that's how multitasking operating systems protect applications from accessing each other's memory. However, inside a program, the applications are limited. You can't put every variable of a program in its own zone surrounded by inaccessible zones. Well, you can, and some debugging programs do it, but that's very inefficient, because the granularity of the address translation is coarse (4kB on most processor architectures). And it doesn't solve all errors. If you have a 64-byte buffer followed by 4032 bytes of unused memory and followed by 4096 inaccessible bytes, an overflow into the 65th byte won't be detected (all you can ensure is that it won't corrupt other variables of the program); an overflow into the 4097th byte will overwrite something else. If you free an object and reuse its memory address for another object, but some part of the program still references the old object, that won't be detected either. (You can't keep addresses of old objects invalid forever, because you'll run out of addresses eventually.)
Many types of misuse of memory can be detected automatically, but runtime is not a good way to do it. You can catch errors sometimes — in addition to accesses to inaccessible memory, you can put explicit checks in your program or runtime environment. For example, you can attempt to detect a stack overflow by putting a known value at the bottom of the stack (a canary) and checking it from time to time. This does help against accidental errors, if the canary value is checked before damage can happen. When it comes to malicious attacks, canary values can make it more difficult to write an exploit, but they don't prevent everything. There are more complex methods of runtime checks, such as taking checksums of parts of memory and arranging to abort if those checksums don't match at a later; those are mostly used in hostile (“grey-box”) environments where the attacker has some limited means to perturbate the execution of the program.
The right time to detect misuse of memory is at compile time. There, it's possible to detect buffer overflows by ensuring that, for every instruction that accesses an element of an array:
- either the compiler can prove that the index is always within the bounds of the array;
- or there is a range check on the index immediately before the access to the array element.
Buffer overflows are only one of the problems. Another common problem is when a program frees the memory allocated to an object, but keeps a pointer to this object around and attempts to access the object later. There is a solution for this, but it has a strong implication on the language design. It implies that the language must forbid programmers from storing pointers in uncontrolled ways. There are several approaches:
- the programming environment invalidates all pointers to an object when its memory is freed;
- the compiler refuses to compile a program that frees an object when there are still pointers to this object around;
- the compiler defers requests to free an object until it can prove to its satisfaction that there no pointers to this object remain.
Most programming languages solve these issues through some form of automatic memory management, where requests to free an object (if they exist at all) are not processed immediately, and all pointers are under the control of the compiler so that it can know what object this pointer is accessing. For example, the compiler sees that an instruction is accessing the 42nd byte of some object A, so it emits an instruction to check that A has at least that many bytes, and it records A as being in use up to that point so that A won't be freed earlier. The most common technique for automatic memory management is garbage collection: the runtime environment has a way to explore all the pointers of the program (which implies that the programmer can't make up a pointer in some hidden way), and it only ever frees objects when there is no pointer to them that the program can accesss.
A majority of programming languages out there use garbage collection and mandatory boundary checks (the two usually go together): C#, Java, JavaScript, Perl, PHP, Python, Ruby…. But the exceptions include some very common languages: C and C++. These languages are designed to give a lot of control to the programmer. They are not compatible with an approach where the programmer is forbidden from making up pointers.
Why do people program in such error-prone languages? Some of the reasons include:
- The weight of history: programs are written in C, and they keep evolving in C.
- Performance. Real or perceived. Automatic memory management has a runtime cost (but then, so does manual memory management).
- The weight of history: programmers know C, so they write more programs in C.
- Overconfidence. Many programmers believe that since they are smarter than a computer, the computer should not override their decisions. (I am smarter than the guard railing on my balcony; so why does it prevent me from walking off the 10th floor?)
- The weight of history: there's a C compiler for every platform. Other languages tend to be less portable.
Languages where the programmer can't make up pointers turns programming errors
that would lead to arbitrary code execution due to a buffer overflow or access to a freed object into denials of service: the program may abort due to an uncaught out-of-bounds exception, or it may run out of memory, but at least it won't do the attacker's bidding. So they don't completely solve the problem — there's no miracle cure — they merely reduce its impact.
While languages with automatic memory management are a step up, they don't fully solve the original unsolvable issue, which is that memory is used for a different purpose from what it was intended for. They're only a step up, in fact. Instead of misusing bytes in memory (which is what buffer overflows can do), attackers may misuse strings: a string that should have been interpreted only as text intended for human beings, is instead interpreted as instructions in some programming language. For an example, an SQL injection attack consists of inserting a single quote in a string which results in part of that string being interpreted as SQL rather than, say, somebody's name. There are ways of relying on the compiler to detect such misuse, but most popular languages don't implement them, and most programmers have no idea that such techniques even exist (see Handing untrusted string input in print function and avoid buffer overflow for an idea of how it's done).
