9

The question is almost completely answered. However, more details are still needed. See Update 2 down here.

I've learnt that the ATA Secure Erase is uncorrectly implemented in SSDs (sources are down here), but I'm still willing to find a way to erase as much as I can on them.

What I intend to wipe is the whole SDD, which includes:

  • The cells that users may access
  • Bad/unmapped/corrupted sectors
  • The over-provisioned space
  • The trimmed cells
  • The Device Configuration Overlay (DCO)
  • The Host Protected Area (HPA)
  • And everything I've forgot

I know that encryption is the best way to simulate a limited "Secure Erase", but before doing that for my new datas, I want to at least make a single pass in order to wipe as much as I can of the old ones.

As I know so far:

  • ATA Secure Erase: Not reliable, still wipe correctly HPA or DCO ?
  • dban: Do not erase remapped sectors, nor HPA or DCO
  • nwipe: Same problems as dban since it's a fork
  • dd: Same as dban and nwipe, but also blocks everytime it meets a bad sector
  • shred: Recommended for files, works like dban, may have issues with SSDs
  • badblocks -w : Should check every sectors destructively, is it correctly implemented for SSDs ?

For now, the best I can do is a badblock -w.

So the question is: What tools can I use in order to erase data as much as possible on an SSD ?

The idea is, based on information I may not know -yet- but you do, to find the most suitable tool listed here, or any other tools not listed here.

Also, anything that may lead to correctly access or/then then delete an SDD's DCO or HPA is ok too. -> This was almost completely answered by @guest, but see update 2.

Same goes for remapped/bad/unnmapped sectors, trimmed cells and over-provisioned space.

Destroying the drive is not an option.

Sources:

Update: badblocks -w may not be reliable (https://lime-technology.com/forum/index.php?topic=23792.0), but I need to dig that up more, unless someone provide an answer here first, which I'm also interested in.

Update 2:

Now, the remaining thing I need to know is: Does the implementation of DCO and HPA respective erasure and disabling is effective -and not badly done like it is for ATA secure erase- ? Furthermore, a naive question here: Does disabling the HPA means this latter will get erased too ?

PS: Sorry if I don't answer right away, I'm working and travelling around the world -thus making this post related- and I often face time-squeezing business. But, I will definitely answer back for sure.

  • 5
    Do you still need the SSD to work afterwards? – Philipp Dec 21 '16 at 9:23
  • theory aside, filling and emptying multiple times should squeeze out old data. – dandavis Dec 22 '16 at 3:32
  • The problem is that it'll make the cells wear out very quickly. And you can't be sure on how much minimum passes you'll need to achieve that, including over-provisionning, remapped sectors, trimmed cells, etc @Philipp: The answer is definiely yes. – X.LINK Jan 15 '17 at 16:56
  • I'm skeptical of the "ATA Secure Erase is uncorrectly implemented in SSDs". Can you provide evidence of a claim for all drives? What about enterprise drives? – dark_st3alth Jan 15 '17 at 18:15
  • I've added sources about SSDs' bad implementation of ATA Secure Erase down the original post. Some SDDs do implement it correctly, but there's no way to know which one does without an extensive research. They also don't tell exactly which drives were tested. In the end, I simply consider all SDDs not able to effectively perform any ATAs' commands (Secure Erase, HPA disabling, etc) until proven wrong. On the other side, It would be naturally mandatory for enterprise SDDs to perfectly implement ATA's commands, but I'm only working with consumer grade SSD and still have my doubts about them. – X.LINK Jan 15 '17 at 18:38
1

Assuming you can set up encryption before first use (and then lose the key to effect a wipe) it doesn't matter if you delete everything. It's completely irrelevant.

If you already have data on the device but it isn't encrypted, then it's too late for that. As you pointed out the tools available all have flaws (through wear balancing, blocking etc) so if the data must be securely wiped your only reliable option is to physically destroy the SSD.

So I'd suggest an industrial shredder (other destruction methods are available)

  • What about installing a new OS on the disk with full disk encryption and fill it? – Mr. E Dec 21 '16 at 14:32
  • No. TRIM and other features move data away from the space that can be controlled by a user. When a cell may soon go bad (OFTEN), its contents are copied to a new one, a table is updated to point to the new cell, and the contents of the old one are left intact. With a crypto-volume, often you need only change the key, and subsequently rigorously scrub the part of the disk that contains the key. The rest can be written over a few times with random data. EMFM does not work on flash memory. – user400344 Dec 22 '16 at 15:21
  • Mr.E: That won't work because it won't cover EVERY bits of the hard drive :/ @Rory Alsop: The goal is to delete unencrypted datas that were already there first. And use the drive again. Destroying the drive is not an option at all. – X.LINK Jan 15 '17 at 17:02
  • 1
    In that case, no - you actually have no way to be 100% sure that all data is wiped. Sorry. – Rory Alsop Jan 15 '17 at 17:09
  • @RoryAlsop - What open source tools are available to attest to secure destruction and the inability to recover data? – Motivated Oct 22 '18 at 2:27
9

I don't know why you say that SSDs implement ATA Security Erase improperly. Modern ones implement it using SED (which uses an encryption key known to the drive and stored in non-volatile memory for all your data, allowing it to simply erase the key to instantly render data unreadable), while others mark all sectors for being TRIMed, which should effectively wipe the data when the garbage collector gets to them. I'm sure there are some out there which improperly implement it (it's not particularly uncommon), but the same applies to hard drives.

It is true that external USB SSDs and HDDs often have problems with ATA Security Erase, but this is due to problems with USB interfaces not playing nicely with the ATA standard. Additionally, USB flash drives (I wouldn't call those "SSDs") don't implement ATA Security Erase, due to having extremely simple firmware and very low-end microcontrollers (compared to SSDs and HDDs, which have very powerful processors, often dual core, sometimes even quad). An SSD is a full computer connected an FTL and an array of flash chips. A flash drive is a microcontroller with flash memory built in.

As for wiping with userspace tools, none of those will affect the DCO, and none of those will directly affect the overprovisioning space. Only using ATA Security Erase will you be able to erase all of that. It's designed to wipe reallocated sectors (as long as they are not too badly damaged), as well as the HPA. The DCO is just configuration data, so it won't contain anything now that it didn't contain when you got it from the store. If you really want to erase it, you can reset it to factory defaults using the Linux hdparm tool, which has a command to restore the DCO.

Note that badblocks -w writes a predictable pattern, and some SSDs will detect this and compress it, resulting in not actually writing the entire thing to the drive. For data destruction purposes, this is next to useless. You will need to write a random pattern to prevent this.

Just use ATA Security Erase. If you know for a fact that your specific SSD firmware has a buggy implementation, and you can't upgrade your firmware to fix it, then use hdparm to send the TRIM command to all sectors, and use dd (or a similar command like ddrescue or dcfldd) to write random data to your drive, and do this several times. Doing it several times is needed not because there is a way to recover deleted data, but because the overprovisioning space cannot be written to in one pass.

You won't be able to wipe completely damaged sectors. If the drive detects that a sector is taking longer than normal to return data, the data being returned doesn't match the internal checksum, or it has to retry too often, then it'll move it to a known-good sector and mark it as bad. It'll do the same if a sector has simply been written to more than a certain number of times (as they fail after a rather predictable number of erase cycles, often around 2000). The drive can be coaxed into manually marking the sector as good again in order to wipe it or put data onto it (which ATA Security Erase does, and which hdparm can do), but that's only if it was marked bad while it was merely losing health. If the sector was so badly damaged that it cannot return any data, and writes stall indefinitely/fail completely, then there's nothing you or the drive can do. The sector is unreachable without you (or an advanced forensic investigator) examining it physically. You can determine if there are any damaged sectors by using the smartmontools package:

# find out how many sectors are reallocated, and how many reallocations occurred
smartctl -a /dev/sda | grep -i reallocated

An example, if you cannot use ATA Security Erase, to wipe /dev/sda as securely as possible:

# find our max sectors (geometry = 91201/255/63, sectors = 1465149168, start = 0)
hdparm -g /dev/sda

# disable the HPA
hdparm -N p1465149168 /dev/sda

# trim every sector in the drive, from 0 to the max
hdparm --trim-sector-ranges 0:1465149168 /dev/sda

# create a crypt device with a random key, and overwrite it a few times
cryptsetup open -M plain -c aes -d /dev/urandom /dev/sda wipe
for i in {0..6}; do dcfldd bs=256k pattern=0$i of=/dev/mapper/wipe; sync; done
cryptsetup close wipe

# restore DCO to factory defaults (this can mess up the SSD, so do it last)
hdparm --dco-restore /dev/sda

But I would still recommend you use ATA Security Erase instead, if you can:

# initiate an ATA Security Erase which can't be disabled until it completes
hdparm --security-set-pass NULL /dev/sda
hdparm --security-erase-enhanced NULL /dev/sda

And next time, encrypt your data!

  • 2
    Thanks for this very instructive post ! ATA Secure erase is improperly implemented due to these studies: cseweb.ucsd.edu/~swanson/papers/Fast2011SecErase.pdf (3.2.4 point) and nvsl.ucsd.edu/index.php?path=projects/sanitize Also, I've mostly heard that the recognized -and useless- pattern is 0, while I haven't heard anything about the other ones, do you have any source about these latter ? Fortunately, badblocks also uses 3 other patterns than 0, but I intend to use a single pass only to not wear out the cells too quickly. But which pattern one is not recognized ? – X.LINK Jan 15 '17 at 17:13
  • 1
    On the other side, I can't know if my firmware does or does not correctly implement ATA Secure erase, so I'll assume it doesn't. Thus I'm finding a way to delete whatever it's still possible to delete, which you provided. Also, does disabling the HPA also means the command will make an effective erasing of it ? – X.LINK Jan 15 '17 at 17:24
  • 1
    "Disabling" the HPA just marks the area the HPA resides as part of the normal drive, so it is no longer hidden and protected. But as the answer says, you will need to overwrite multiple times to increase the probability of overwriting all or most of the overprovisioning space. – forest Sep 24 '18 at 14:01
  • @forest - If a drive is encrypted prior to HPA being disabled does this pose a risk to data leakage or remanence? Does HPA need to be disabled to enable effective encryption of drives? Does DCO need to be reset? – Motivated Oct 22 '18 at 2:23
  • 1
    @Motivated The HPA does not need to be disabled just to encrypt the drive. As long as the only things you put in the drive are encrypted, then the HPA isn't relevant. And unfortunately there are no reliable methods to erase data on a flash drive without physical destruction. – forest Oct 22 '18 at 2:58
2

Short answer: Disabling the HPA and resetting the DCO only unhides previously reserved sectors on the drive, and does not involve any secure erasure itself. It only makes these sectors available to the system for later deletion by system tools.

Update 2:

Now, the remaining thing I need to know is: Does the implementation of DCO and HPA respective erasure and disabling is effective -and not badly done like it is for ATA secure erase- ?

You can disable the HPA (i.e. set its size to 0), and you can reset the DCO to factory defaults. Neither of these involves erasure. Unlike the HPA, the DCO cannot contain personal information (whereas the HPA may contain data that was deleted but not overwritten). The DCO is just a small amount of configuration data. If it was even possible to wipe it completely, you would brick your drive. All you can do is reset it to factory defaults. You don't need to worry about securely deleting it, because nothing will have been written to it since the day you bought it from the store.

Example of the information stored in the DCO, to show how little it actually stores:

# hdparm --dco-identify /dev/sda
/dev/sda:
DCO Revision: 0x0001
The following features can be selectively disabled via DCO:
    Transfer modes:
         mdma0 mdma1 mdma2
         udma0 udma1 udma2 udma3 udma4 udma5 udma6(?)
    Real max sectors: 1465149168
    ATA command/feature sets:
         SMART self_test error_log security HPA 48_bit
         (?): selective_test conveyance_test write_read_verify
         (?): WRITE_UNC_EXT
    SATA command/feature sets:
         (?): NCQ SSP

Furthermore, a naive question here: Does disabling the HPA means this latter will get erased too ?

No. Think of the HPA as a hidden partition. Configuring the HPA is akin to resizing this reserve partition. Disabling the HPA is setting the size to 0, allowing your operating system to now see and use the space that was once reserved. Once its disabled (set to 0), that previously-untouchable space can now be subject to erasure using system tools. But the only way you can interact with the HPA that matters is by setting its size.

Note that the DCO is also capable of hiding data by making the drive present itself as smaller than it actually is, although in a different way than the HPA does. Some vendors will intentionally reduce the size through the DCO in order to make one drive size, cripple it by limiting the size with the DCO, and sell it at a reduced price, selling the full-sized version (i.e. the version without DCO limitations) at full price. You can remove any potential limitations by resetting your DCO to factory defaults before performing any wipe, although beware that any low-level configuration changes to a drive carries risks

Regarding your comment on my first answer (I can't reply in the comments since this is a guest account), the paper you linked seems to be talking about older SSDs which improperly implement the ATA Security Erase standard. As I said, it's not particularly uncommon for the firmware to be written poorly enough to fail at implementing it. However modern drives all use SED, which negate the issue of flash cell memory remanence. Overall, a modern drive is likely to be more reliable, by far, than badblocks, by orders of magnitude.

A simple test would be to overwrite the drive with a pattern like 0xdeadbeef, initiate ATA Security Erase (with a SED-enabled drive), and check to make sure that the pattern is nowhere to be found. With this, you can know if it's been implemented properly. If the command returns almost instantly, and the pattern is gone, then SED has very likely been implemented correctly.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.