I'm not entirely sure why you are worried about protecting an AES key, since an attacker who observes you using that key can probably also see your secrets. But I will continue, assuming symmetric cryptography is not an option.
What you are describing aligns perfectly with a use case for public key cryptography. Each family member that you trust generates a keypair and shares the public key with you. You then use all of their public keys to encrypt your secrets.
I don't think "sharing" and "private key" should be in the same sentence, even in this case. So ideally, they would generate their own keypair and you would never have access to it.
However, I imagine many family members would not be comfortable using PGP or the like, so there definitely could be usability issues with this method.
There are tools out there specifically meant for sharing passwords or other simple text secrets among a team. For example, there is the Unix pass
tool that uses GPG and Git to share and store passwords among multiple users, who can be added or removed over time. There are many different frontends for this tool out there, some likely a bit more user friendly.