You seems to have a wording problem with the delete term and a wrong expectation about what the functionality should do.
You can check the simple definition on the Merriam-Webster website:
delete: to remove (something, such as words, pictures, or computer files) from a document, recording, computer, etc.
The goal of the delete feature is to remove the selected objects from their current location. They can be moved to another temporary location (the trash bin) to prevent any accidental loss, or the space they occupied might directly be marked as free.
Compare this to the definition of erase for instance:
erase: to remove (something that has been recorded) from a tape (such as a videotape or audiotape) or a computer disk; also : to remove recorded material from (a tape or disk)
: to remove (something written) by rubbing or scraping so that it can no longer be seen
: to remove something written from (a surface)
This one really goes to a lower level, here we are not just removing the file from the folder, we are removing the file's data from the disk, and in this subtlety lies all the difference.
Erase is the word you may most encounter in Windows world, in Unix world you may encounter wipe instead which describes the process from a more technical point-of-view:
wipe: to clean or dry (something) by using a towel, your hand, etc.
: to remove (something) by rubbing
: to move (something) over a surface
Why don't the OSs do this by default? For several reasons:
Common home users do not need such feature, all they need is either to simply remove a file from a folder, or in the case of external drives ensure that it gets cleaned completely ("slow" format in layman terms). They rarely need to erase a single specific file.
Common home users actually do not want such feature. You will see that the default is not only to not erase the file, but not really delete it either: instead it is moved to the trash can because home users expect their OS to be able to save them in case of a wrong manipulation. You often have to use a specific key combination (Shift + Del option) to delete a file without going through the trash can.
Finally, such an option may not be easy or even possible to implement. Technological evolution has added several logical and physical layers around your actual data:
Some OSes take regular snapshots of the file system content, in such a case your file might be included in one or several of these snapshots. It may become complicated to ensure proper file erasing in such a condition without endangering the snapshots integrity.
As mentioned in other answers and comment, to increase you storage device life there is an abstraction level between the actual storage and how it is seen by the OS. In such case, while the OS can do its best to ensure that the file's data gets deleted at the file system level, it has no way to ensure it has been effectively erased at the storage level.
On one side you have the OS which reasons in terms of files (to simplify) but does not really care about byte storage, on the other side you have the storage device's firmware which reasons in terms of bytes but does not know anything about the files and how the file system is actually set.
So, does this means that everything is lost and we need to wait for some improbable future to bring us file system implemented natively in the storage device's firmware? No, but what you should do depends on your actual concern:
If your concern is about the data at your file-system level, then you have third-party software available which will add you a new erase / wipe option to your contextual menus. You need to manually ensure though that copies of this files are not present in any file-system snapshot and backup, since this require a case-per-case decision the OS cannot do it for you (you do not want your OS' eraser wizard to screw your backup or the sake of helping you, do you?).
If your concern is about the data physically stored on your storage device, use file-system encryption. This will ensure that potential "leaking" data due to wear leveling and bad block handling will not be exploitable by anyone getting their hands on your storage device.