I'll argue to delete them. The only thing that prevents these accounts from being used is:
Inactive administrators cannot log in.
And that is a defense against brute forcing (or fooling) an authentication, yet it does not defend against privilege escalation.
OWASP guidelines for web applications argue that a client should never hold a session with authentication for two separate accounts. That would prevent privilege escalation in most cases. Yet, by far, not all web applications follow OWASP guidelines.
Therefore, keeping those inactive but still privileged accounts is a possibility of an attack vector.
The other side of the coin is that the procedure for deleting the accounts may consume resources or, even, be insecure in itself. But managing to produce an account deleting procedure that is vulnerable to an attack is much less probable than being hit by a privilege escalation.
Just don't make the periodic account deletion procedure to delete the inactive accounts by first logging into an active administrator account over the network.