Suppose I have N users who should be able to decrypt a subset of documents D1...Dk. N is very large and users are often being adding and removed so it is impractical to encrypt a given document Di using all of their public keys.
As I've read, a common solution here is to create a group G with its own key pair, and its public key is used to encrypt the document.
What I can't figure out is how to give all users coming in and out of G the ability to use G's private key without storing it on the server.
I thought about encrypting G's private key with the public key of all its members, but what happens when a new user joins? I feel as though I would need to encrypt the private key again, this time with the new user's public key mixed in, however as the server I no longer have access to the unencrypted private key.
The closest solution I've seen is to create a new key pair for G and re-encrypt Di, but for a large number of documents this too becomes very impractical.
I should add that the system shouldn't be allowed to decrypt user's documents.
Is there some pattern using PGP (or any other standard really) that can help me out here?