The main reason malware can't evade AV this way is because the on-access scanner will catch that.
From the AV point of view, malware moving to different files is the same problem as new files being created during the scan. A very simple solution is for the AV to keep track of new files being created (which includes current files being renamed). This is done by hooking the OS API or other lower level hooks.
In theory, a perfect rootkit will be invisible to an AV. To increase AV chances against rootkits, the AV starts very early in the startup sequence, it uses low level ways of communicating to the OS and hardware, and it will monitor tampering activity on itself. Also, the first stage of a scan is on active processes and system binaries. That's done especially because in-memory malware can play tricks like you mentioned.
Also bear in mind that any strange activity on the side of the malware, like moving files, jumping through the memory of different processes and using rootkit-like methods will only attract attention to it because legitimate software will not have such behaviour. A strange behaviour will be flagged by the AV heuristics and that might help to detect the malware or the binary might be uploaded to the AV servers for further analysis and cross-checking.