Let's assume we have a system delivering files (such as updating a program) that are digitally signed with a trusted CA. If an attacker somehow manages to install custom certificate as a trusted source on victim's machine, and then man-in-the-middle changes incoming files to other signed with this custom certificate, is there a way to prevent this? For example: can you check against only a pool of trusted-by-developer certificates?
Yes, this is called certificate pinning. You would simply verify the signature against a particular public key rather than any trusted public key when doing the verification of the signature.
That said, if the attacker has already compromised the local machine and installed a root certificate, they are unlikely to need to mess with your update mechanism to compromise your software anyway since they are already performing trusted actions on the client.