There were many systems... you can find some references from the Wikipedia page. The core idea is always the same: make a "special" floppy disk such that it exhibits a peculiar behaviour that can be tested for, but not reproduced, with stock floppy disk drives. For instance, make tracks a bit thinner than normal, which allows you to put a bit more data on the floppy; a "normal" drive should succeed at reading the extra tracks (still in the physical tolerance) but not writing the same setup on a brand new disk. Another method would be to have sectors with bad data (checksum mismatch).
Such kind of physical methods were easier to pull off in the 1980s, when you could rely on specific hardware features. An Atari ST was always an Atari ST from Atari, not a clone with possible distinct hardware and thus subtly different behaviour when facing "anomalous" floppies. The openness of the PC architecture mostly killed off copy protection scheme based on weirded up floppies.
The main problem with all such methods is that the corresponding check must be done in some place in the software, so cracking the copy protection scheme usually meant finding the relevant piece of code and changing a conditional jump opcode with a nop (or an unconditional jump). The essence of all these copy protection schemes is security through obscurity, which does not work well, especially when the source element is sold by the thousand and the attackers are schoolboys with too much time on their hands.