Why didn't OSes securely delete files right from the beginning? And why do they still not do this?

Question

After decades of hearing that "delete" does not really make the data impossible to recover, I have to ask WHY the OS was not corrected long ago to do what it should have been doing all along? What is the big deal? Can't the system just trundle along in the background over-overwriting and whatever else has to happen? Why do we need additional utilities to do what we always thought was happening? What is the motivation of OS developers to NOT correct this problem?

ADDITION: This is not a technology question, because clearly it IS possible to delete things securely, or else there would not be tools available to do it. It is a policy question: If some people feel that it is important and should be part of the OS, why is it not part of the OS? Many things have been added to OSes over the years, and this could certainly be one of them. And it IS an important issue, or there would not have been articles and stories about it for about 3 decades now. What is with the inertia? Just do the right thing.

Answer 1

Instead of another "You are wrong because" answer I'd like to take a slightly different approach:

Early computer OS's were written by programmers for programmers. Any one who programs and knows what pointers are understands that "deleting" a pointer doesn't delete the thing its pointing at: they are separate.

That doesn't mean that delete doesn't actually delete. That pointer is gone. Trying to use it after "deleting" it (freeing the memory, rebinding the name) can result in bad things happening.

But history marches on, and now end users who have a different concept of delete (like yourself) are in the picture. They (and you) have expectations that are not unreasonable (whatever else is said in this thread).

But delete will not ever mean (for a computer) what you think it should: there are reasons both technical (detailed quite well in other answers) and social (45 years of inertia).

The modern (and I'm including *nix) OS abstracts a lot of things for you: you no longer need to be a computer expert to own/operate a computer in the same way you no longer need to be a mechanic to own/operate a car. The price you pay is that those abstractions are leaky: there's a fundamental disconnect that can never quite be bridged. A computer "document" isn't really a document, a "desktop" is not a desktop, a "window" is not a window, etc.

Answer 2

Because of the following reasons:

• Performance - it takes up resources destroying files. Imagine an application that uses hundreds or thousands of files. It would be a huge operation to securely delete each one.
• Extra wear and tear on the drives.
• Sometimes the ability to retrieve a file is a feature of the OS (e.g. Trash, Recycle Bin, Volume Shadow Copy).
• As noted by Xander, sometimes the physical storage mechanism is abstracted from the OS (e.g. SSDs or network drives).

Answer 3

It doesn't have to be corrected because it's not a fault.

The pointers to the file are deleted, and the area the file occupied is marked as free space. The drive then overwrites this area in its own time. It's purely there to save wear and tear on the drive. After all, storage devices (especially SSDs) have a limited number to times they can write before they fail. Most users would not appreciate their drive failing after 6 months.

Secure solutions do exist with tools available to securely wipe free space on a hard drive.

Answer 4

You seems to have a wording problem with the delete term and a wrong expectation about what the functionality should do.

You can check the simple definition on the Merriam-Webster website:

delete: to remove (something, such as words, pictures, or computer files) from a document, recording, computer, etc.

The goal of the delete feature is to remove the selected objects from their current location. They can be moved to another temporary location (the trash bin) to prevent any accidental loss, or the space they occupied might directly be marked as free.

Compare this to the definition of erase for instance:

erase: to remove (something that has been recorded) from a tape (such as a videotape or audiotape) or a computer disk; also : to remove recorded material from (a tape or disk)

: to remove (something written) by rubbing or scraping so that it can no longer be seen

: to remove something written from (a surface)

This one really goes to a lower level, here we are not just removing the file from the folder, we are removing the file's data from the disk, and in this subtlety lies all the difference.

Erase is the word you may most encounter in Windows world, in Unix world you may encounter wipe instead which describes the process from a more technical point-of-view:

wipe: to clean or dry (something) by using a towel, your hand, etc.

: to remove (something) by rubbing

: to move (something) over a surface

Why don't the OSs do this by default? For several reasons:

• Common home users do not need such feature, all they need is either to simply remove a file from a folder, or in the case of external drives ensure that it gets cleaned completely ("slow" format in layman terms). They rarely need to erase a single specific file.

• Common home users actually do not want such feature. You will see that the default is not only to not erase the file, but not really delete it either: instead it is moved to the trash can because home users expect their OS to be able to save them in case of a wrong manipulation. You often have to use a specific key combination (Shift + Del option) to delete a file without going through the trash can.

• Finally, such an option may not be easy or even possible to implement. Technological evolution has added several logical and physical layers around your actual data:

  • Some OSes take regular snapshots of the file system content, in such a case your file might be included in one or several of these snapshots. It may become complicated to ensure proper file erasing in such a condition without endangering the snapshots integrity.

  • As mentioned in other answers and comment, to increase you storage device life there is an abstraction level between the actual storage and how it is seen by the OS. In such case, while the OS can do its best to ensure that the file's data gets deleted at the file system level, it has no way to ensure it has been effectively erased at the storage level.

On one side you have the OS which reasons in terms of files (to simplify) but does not really care about byte storage, on the other side you have the storage device's firmware which reasons in terms of bytes but does not know anything about the files and how the file system is actually set.

So, does this means that everything is lost and we need to wait for some improbable future to bring us file system implemented natively in the storage device's firmware? No, but what you should do depends on your actual concern:

• If your concern is about the data at your file-system level, then you have third-party software available which will add you a new erase / wipe option to your contextual menus. You need to manually ensure though that copies of this files are not present in any file-system snapshot and backup, since this require a case-per-case decision the OS cannot do it for you (you do not want your OS' eraser wizard to screw your backup or the sake of helping you, do you?).

• If your concern is about the data physically stored on your storage device, use file-system encryption. This will ensure that potential "leaking" data due to wear leveling and bad block handling will not be exploitable by anyone getting their hands on your storage device.

Answer 5

For performance reasons. Deleting the file from the index, and declaring that the zone where the file was is now free and can be re-used is far more efficient that erasing all data over that zone.

Answer 6

Not everybody agrees with your definition of what "delete" should mean.

For 99.9% of users, they're not worried about someone sniffing around getting data. They want space to store more torrented Teletubbies episodes. For most people, simply no longer having reserved the space for the file is sufficient.

Then there's the group of individuals you are a part of, that want to have the file erased so that it cannot be recovered. However, did you properly consider the cached copies of the file that may be elsewhere on your harddrive? It can actually be difficult to ensure something is fully deleted. For someone who wants a feature like this, you may have already balanced the security questions, but do you think the 99.9% did?

Finally, consider those who really care about erasing their secrets. The government standard process is to throw the hard drives into a grinder. Why? Did you know that when you erase the file, some of the magnetic properties of the disk keep your data accessible? It can't be accessed by the normal hard drive head, but take the platter out and put it in an expensive magnetic reader, and they can actually pull data off.

If the government wishes to reuse the hard drive, the standard process is a 7 fold wipe, where you write to every sector of the disk 7 times. They usually just use the grinder, because this process is so mechanically intensive that most disks don't even survive the encounter.

What does 'delete' mean anyway?

EDIT: I tried to draw a few points, and let others draw the lines, but it looks like it might help for me to draw the line too. Security always involves compromises. Few people understand security well enough to pick the level of compromise properly. For those who can, you can then divide people into three categories. There's one category of people who are less concerned with security, and will hate a product for pushing more security than they want at the price of usability. There's one category of people who are more concerned with security than you are, and will be frustrated with the lack of security in the product. Finally, there is one category with you, who is happy with that balance.

If everybody has their own definition of what "secure enough" is, getting the secure delete files "right" from the beginning is harder than it sounds.

Answer 7

For much the same reason as real-world trash collectors don't shred everything before throwing it into their truck.

There are two primary reasons why people may get rid of objects or information:

1. They wish to prevent their use by anyone else (or perhaps some specific person or people).

2. They do not believe that the items would have sufficient value to anyone as to justify the expense of keeping them or of finding someone to whom they might be useful.

Those who are getting rid of things for the first reason should destroy them so as to render them unusable, but for those who are getting rid of things for the second reason, it's preferable to minimize the total remaining expenses associated with the items in question. Because computers generate a lot of information which will become useless after a relatively short time, and because--for most computers--much of the information would be unlikely to cause any damage to anyone if released, there is in many cases little benefit to deliberately destroying information. It is sufficient to ensure that any space which the information had occupied will be available for reuse if and when a need for it arises.

Answer 8

Actually you're messing up two different procedures : the deletion and secure wiping. The first one is just a standard FS-dependent operation, a determined and standardized one. The second - the secure wiping - has a lot of approaches, unerase-protection grades, standards, etc. Because of that fact, it is logically correct, that if you need the second category of one, you're responsible to select and activate a mechanism suitable in your very own unique situation. There's no common and standardized in every case/task/environment implementation for secure wiping.

Answer 9

Why didn't OSes securely delete files right from the beginning?

You mean from the beginning of the deletion, or the beginning of operating systems? Early operating systems were trying to focus on technological advancements like storing stuff on hard drives rather than floppy drives or punch cards. There was no need for the operating system to do anything special to make it impossible to retreive data from floppy drives. (Sorry, just having a bit of fun poking at the reliability of floppy drives there.) Back in those days, the ability to "save" was a big deal. To do file maintenance, like being able to "delete", might be a buried-and-rarely-used menu option. To "securely delete" would seem like nonsense to people who could just throw the disk away, or burn it first. Why use the noisy floppy drive, which had much lower speeds than we are used to today, to wait for a half-minute or longer just so it could overwrite ones with zeros? In the day when nobody understood what a "double click" was, because they didn't even have a mouse and so they never even made a single click, and they thought that "copy and paste" was super advanced technology, they getting people to even understand the purpose would have been quite challenging.

Then, when hard rives came along, the operating systems simply did the straightforward thing, which is to use the same computer code that has been proven reliable for floppy drives.

And it IS an important issue, or there would not have been articles and stories about it for about 3 decades now.

Actually, this may be more of a reflection on Journalism's goals to write things that people may find interesting, or it may be the result of educational efforts. A lot of people simply don't understand that deleting is rather non-permanent, and it is good for people to understand things, so education is a positive thing. This doesn't mean that softare's default behavior should be changed.

If some people feel that it is important and should be part of the OS, why is it not part of the OS?

Because some people feel like it is not important. Although wiping might be much more useful for some people (who are, by all means, free and welcome to wipe (instead of delete) if they wish to do so), different people may have different priorities. I, for one, delete stuff so that I have available disk space instead of used disk space, and speed up actions that affect all files (like copying all of the used disk space on a hard drive), and so I don't need to see filenames of old files, and so that I don't accidentally open an old file when I want a new file. Deleting is an effective way to hide unwanted data (like old data) from software so that software doesn't use that unwanted data. Before I started to back up data more reliably, I actually appreciated the occasional ability to use recovery software that could "undelete" data. As others have expressed, wiping can cause additional slowdowns and additional "wear and tear". Since my computers haven't been in the presence of people who I am concerned may dig through my deleted data in order to try to learn something that I hoped to be secret, wiping has never been a huge priority for me. With all that in mind, in response to your last statement, “Just do the right thing”, I think that deleting has been the “right thing” for my usage scenarios far more frequently than what wiping would be. I think the biggest simple reason, which answers many of your questions, is that identifying which behavior is the “right thing” for most scenarios might not be quite as straightforward as you're giving it credit for.

Answer 10

And why do they still not do this?

The other answers have answered why it wasn't done in the past (performance, user expectations etc.) so I will just add a note on why it still isn't done now. One problem is that secure deletion is only considered a partial security solution: it will delete one copy of the file, but most modern applications will leave substantial traces behind - history (undo) buffers, backup copies, etc. - from which data can be gathered. There is also the issue of files being accessed before being deleted (e.g. you were going to securely delete that file on your drive, but someone stole the drive before you had a chance). For these reasons, modern operating systems have instead implemented complete encryption for user storage, which makes secure delete unnecessary. With encrypted user data, even though the drive retains the image of a "deleted" file, all traces of that file (and every other file) are hidden to an attacker who does not have the decryption key. Chrome OS, Android, and iOS are all now encrypting user data by default, so, on these systems at least, there is little need to securely delete individual files.

Answer 11

Other answers are good and have covered the question fully on a technical level. But I think another answer is prompted that takes a different view of the problem, as (IMHO) this question isn't really a technical one (asking why something wasn't made differently is more of a business decision than technical).

The question asks why the OS doesn't really delete the data. The maker of the OS can make it do whatever they like. There's certainly no reason why someone can't build an OS that can delete the data. But why would they? Try flipping the question around, why would someone build an OS that deletes data when it is easy to download a utility that does it for you? Also, it's usually the philosophy that OSs should be minimalist and only contain features necessary to the operation of the computer.

I'm not trying to be harsh with my answer, what I'm saying is the question seems to be asked from the wrong perspective and there's preliminary pieces of knowledge missing.