I am looking for existing protocols for a group chat with two things:
End to end encrypted. Just what you would expect: messages are only decipherable by the chat members and message tampering is detected.
It should not encrypt each message for each member individually. The Signal Protocol does this, turning group chats into many one-on-one chats, which is not a proper, scalable solution to the problem. With potentially hundreds of members in a group, even encrypting an encryption key for each member is a considerable downside.
Every new member may receive everyone's public key upon joining, and any group key(s) must be rotated when a member leaves. This scales reasonably enough, and there might not be a way around it without compromising security, so this is allowed and does not count as 'encrypting every message for everyone'.
I've looked for existing protocols, but came up with zero results that meet these requirements. I thought I read about something a few years ago where the group derived a common key and used that or something, but I cannot find anything like that.
Tox has some extensive documentation but somehow I can't seem to find how encryption happens in a group chat.
Another source even claims "you can't make groups with end-to-end" (though I am fairly sure they are mistaken).