For the specific situation you are alluding to, using a symmetric key and RSA keys is the right path. Namely:
- Data is encrypted with symmetric key K (with AES or some other algorithm which does the job).
- Every user who should have access to the data has a RSA key pair. Each user has his own key pair, distinct from that of any other user. Let's call Pu the public key of user u (and Su the corresponding private key).
- K is asymmetrically encrypted (RSA) with the public key of each user. This means that the database contains several values EPu(K) (encryption of K with the public key of user u).
- When a user wants to access the data, he uses his own private key Su (from his flash drive) to decrypt his value: DSu(EPu(K)) = K. With K, the user can access the encrypted data.
- When you want to add a new user v, you generate the new key pair for that user (Pv, Sv) with the private key stored on a new flash drive which you give to the user. You also encrypt K with Pv and add the new EPv(K) to the list of such values in the database.
Removing a user is tricky, because of a general fact: you cannot enforce forgetfulness. Any user who had access to the data at some point could have grabbed a copy of the whole (decrypted) data and still remember it. The best you can do is to make sure that the removed user will not have access to the new data added after the eviction of that user. There are thus two methods to revoke user u data access:
The fast but incomplete method is to simply remove EPu(K) from the list of stored values. This will be sufficient to deny subsequent access from user u if that user is honest and did not keep a backup of EPu(K) (or, simply, a backup of K).
The thorough method is to generate a new master key K', decrypt all the data, reencrypt it with K', and encrypt K' with the public keys of all users except u. The database will then contain EPv(K') for all users v except u. This effectively locks out user u from any new data element.
The system described above can accommodate smart cards in lieu of "flash drives". There is not need for any entity other than user u to actually access the private key of user u.
Note: all of the above is about controlling read access to the data. Controlling write access, detecting alterations, duplications or suppressions is another matter.