On ext4 there is a chance that your data (your secret keys) can be recovered irrespective of the underlying hardware, depending on how you mount it. (The following explanations are valid for many other journaling filesystems, too, and you always have a problem with copy-on-write filesystems such as btrfs and others that are snapshot-capable).
The curse of journaling/copy-on-write file systems
This has to do with how modern (journaling) filesystems make sure that if you have a power loss in the middle of writing to the filesystem, the filesystem will still come up in a sane state. With small disks, we had to run programs such as fsck to fix a filesystem after such a crash. With large disks, running fsck would take too long, so journaling filesystems are a much better option.
However, they achieve their recovery magic by keeping a journal of operations (hence the name) they can either replay in full or discard if there is no commit on record (e.g. if your computer died in the middle of writing something to disk).
Now depending on how much your filesystem writes into the journal (just the file metadata operations or also the associated data), you might have a problem.
One way to be able to recover to a clean state after a writing data to a file went wrong / got interrupted in the middle) is to never write data to the same location - so if you overwrite a file's data, the new contents are not stored in the same disk block as the original file contents. A new disk block is allocated instead. This is called copy-on-write (COW for short). This way, the old file contents can be easily recovered when it turns out the write operation didn't finish correctly.
However, this also means that your secret keys still reside on disk after being "overwritten" with new data and can be found by a simple
grep operation on the partition device, for example:
grep "my-secret" /dev/sda1
strings /dev/sda1 | grep "my-secret"
So you don't need to be a rocket scientist to recover previously overwritten data when the file system is mounted with the data journaling option.
data journal on ext3 and ext4
ext3 and ext4 allow you to turn data journaling on (
data=journal option) or off (which, I think, is the default). I think that without data journaling (or at least with
data=writeback), your files should get overwritten in place (that's what
man ext4 suggests, anyway). But I'm not 100% sure, so you better test it.
Problems caused by solid-state disks and wear-leveling algorithms
SSD's and flash drives pose more problems - wear leveling algorithms make it pretty much impossible to know where data physically ends up.
SSD's trim option may help you because it allows the SSD controller to mark blocks as deleted, so the controller can schedule them to be overwritten at an earlier time. Still, you have no guarantees here, either. Cheap SSDs might not even implement it and lie about it to the operating system.
Finally, even rotational magnetic disks pose problems. Most of these disks which are still operational have smart controllers which can substitute a working sector for a defective sector on the fly. This may happen any time, so if you're very unlucky, the sector that contains your secret might end up marked as defective and transparently swapped out. Usually you can't access these defective sectors (they're defective, after all), but that doesn't mean experts with the necessary diagnostic hardware can't still read these defective sectors. However, I wouldn't worry much about this scenario, it's just something to keep in mind.
What can you do?
So your only safe bet is to encrypt the filesystem where you store sensitive information. Then you can make sure it's really inaccessible by deleting the encryption key. This is the only solution that works no matter what kind of filesystem and hardware you use.
Another way to actually overwrite data is to create a file that takes up all the free space of a partition:
dd if=/dev/zero of=stupidly_big_file
However, this may take a very long time and it isn't quite foolproof.