As far as I understand, usually the game anti-cheat runs in another process from the game client. This makes it obvious to try to simulate anti-cheat, for example, to replace it with your program, which interacts with the game client and the game server in the same way as anti-cheat, but does not perform any actual cheating software detection.
Since it is not so difficult to log all interprocess and network interactions of a particular process, it seems a simple (but tedious) task to reverse engineer the protocols used by anti-cheat for communication and write a mock implementation that would look legitimate from the perspective of the game client and server.
What prevents cheat developers from disabling anti-cheat in this way? How can anti-cheat protect itself from such attacks?
The best idea I came up with is to integrate message signing in all (or some of) anti-cheat communications and make it really difficult to obtain the key via obfuscating. But this would only make it harder to crack communication protocols and is vulnerable to runtime memory expection attacks, since anti-cheat have to assemble the key at some point in runtime and put it in the memory