What if the file is a system file?
Before releasing the signatures to public, the antivirus vendors would test them on different machines with different OS and programs installed. If the signatures contain a hash which detect a system file, and is a false-positive, they'll fix the signatures and re-test them and then release to public.
Or if the files are EXE/DLLs required by system to work normally?
The reason a detected file is quarantined is that if it is a false-positive, it can be restored to where it was. Almost all the antivirus vendors provide a feature to users to report potential false-positives. If it is found to be a legitimate error, they would fix it in the next update, and that file won't be reported in the next iteration of updates.
How does an antivirus suite decide whether a file is safe to be quarantined?
If the suite detects it to be malicious by any means(signature hash, heuristic, etc.), it quarantines the file. Now the decision is left to the user if he/she wants to revert it back or not.