I am talking about crypters used to encrypt files like viruses and keyloggers for the sole purpose of bypassing antivirus detection. How do they work?
In some circles, "crypter" and "packer" are synonymous to mean binaries or programs which are self-checking and/or self-modifying. Crypters may, more specifically, mean self-modification that includes encryption and/or code scrambling (see more below).
I suggest that you read Chris Eagle's The IDA Pro Book for an introductory understanding of packers and their disassembly of components and architecture.
Daniel Reynaud's work on packers and self-modifying code is also very fascinating. He makes some distinction between integrity checking, blind writes, code scrambling, and encryption of PE files in this blog post about visualizing packers and self-modifying programs. His crude system of checking for these various technique implementations in packers using dynamic binary instrumentation (not the best way of going about this, but it can be very fast compared to other methods) is available at https://code.google.com/archive/p/tartetatintools/
You could give some of the encoders in the metasploit framework a look. In particular the Polymorphic XOR Additive Feedback Encoder ( Shikata Ga Nai ) might be worth a look, as it is dicussed in the Metasploit Unleashed documentation in regards to bypassing antivirus detection.
We can make a difference between the packers which, from an binary create a new binary, with a different signature which is not comprehensible by a disassembler and which will be unpack in memory for go back to the original code, we call it metamorphism. And there is the packers which from a binary create a new binary with a new signature which can be polymorphic but which will edit his own signature at runtime every time he will get executed, we call it metaporphism.
If you want to see metamorphic engines, you can take a look at my code (only for elf) : AD_1DA
Few cool papers : https://vx-underground.org/papers.html (There are all very cool especially the Infection categorie)