You are entering the murky area of revocation. Basically, you want to have a system which "works" but with a cancel switch which says: even though everything looks legit with cryptography and everything else, don't open.
If your system is completely offline, there is no solution. A "cleaner" has the power to open all the lockers; he cannot lose this power if everything is offline because the information about the hijack of the cleaner state (theft of his keys) cannot enter a totally offline system, since it comes "from the outside". To solve your problem, you necessarily need an at least intermittent network.
The usual solution, for instance in X.509, is to have short-lived signed objects which declare that "for now, everything is fine". In your cleaner/locker analogy:
- Each locker has a clock and knows the current date and time.
- Every morning, the cleaner goes to the main cleaning office and shows his face and keys to an officer who verifies that the cleaner is indeed legit and allowed access; the officer then gives to the cleaner a signed "access token" which says "cleaner X is allowed access up to March 8th, 23:00 UTC".
- When the cleaner wants to open a locker, he must insert his key and the access token. The locker verifies the signature from the officer, and also that the access token is not yet expired.
In X.509, the "access token" is a CRL or OCSP response. Details vary on how the objects are brought to the validator (here, the locker). In the analogy above, the "intermittent network" is the central office; this is the point where the system is transiently "online". Alternatively, signed access tokens could be distributed to lockers proactively by someone else.
A cornerstone is that validators MUST have a reliable time source, for a suitable notion of time (you don't necessarily need much in terms of precision and accuracy).