It is possible for a server admin to silently add an infiltrator to group chats. But without letting the group to be notified, It's not possible well atleast in Signal protocol.
Signal is open-source so you can verify from the source that a message that someone has joined the group is shown on GUI. While what's app if pressurised by the government can choose to program it in a way that no message is shown on GUI if they add an infiltrator.
But that won't prevent the infiltrator from getting detected. It is because the group key exchange would still be visible. To add a member into a group, the group administrator sends a group management message to each group member which instructs them to send their own group key to the person group admin wants to add. Now if you observe the key exchange is already started to happen. This group management message also notifies the members visually so that they know that said person is now part of the group. This is where you see that message on GUI. Once that person is added, he generates his own group key and sends to every member of the group as a personal one-to-one pairwise encrypted message. Yet another time, key exchange can be seen.
The group key is not same for all. Each member has its own group key and everybody keeps list of group keys of each other. That's why it is very important to flush old group keys when someone leaves because former member has learnt group keys of every member.
In Signal app, server doesn't involve with group administration. Everything happens exactly that way except the fact that there is no admin in Signal and every member can send group management message to the group members.
But In WhatsApp, a group admin doesn't send group management message directly to the group members. It should have but they went out of specifications. Instead the group admin sends this message to the server, the server verifies if he is actually an admin and then it instructs group members for key exchange. Actually, now WhatsApp sends group management message to only subset of members but it is enough, other members can send their group keys later. Even if the infiltrator is silently added by WhatsApp server, each member has to be instructed to prepare for the key exchange else the infiltrator won't be able to decipher the messages.
A team of crytopgraphers from Germany’s Ruhr University Bochum say they have uncovered flaws in WhatsApp’s security that could Let an Infiltrator Add Members to Group Chats
The described weaknesses enable attacker, who controls the WhatsApp server or can break the transport layer security, to take full control over a group. Entering the group however leaves traces since this operation is listed in the graphical user interface. The WhatsApp server can therefore use the fact that it can stealthily reorder and drop messages in the group.
In a response to the story, Moxie Marlinspike, co-founder of Open Whisper Systems, which developed the end-to-end encryption used in Signal and WhatsApp, refutes the researchers’ claim that an attacker could conceal alerts from other chat members that someone was added to a group. “The attacker will not see any past messages to the group; those were e2e encrypted with keys the attacker doesn’t have,” Marlinspike writes, adding, “All group members will see that the attacker has joined. There is no way to suppress this message.”
The attacker has joined message cannot be dropped because it is just not a message on GUI, it's an instruction for key exchange.
But let's suppose that the message was dropped and the infiltrator has joined the group without triggering any alert. Now if the group starts talking in his presence, all the infiltrator can see encrypted messages that are encrypted by the group key he never had which is same as sitting on the server and watching the passing messages. Without participating in the key exchange, group keys cannot be shared.
It doesn't matter how apps handles group management. If they are using E2E, a key exchange is always visible before communication can take place.
Theoretically it is still possible to verify if someone has been added without being notified by verifying whether a new group key is added. The group keys are only added or removed when someone leaves or enters the group.
if there is a backdoor added by the developer, then the app is compromised anyway so it is just not limited to silently adding an infiltrator. An adversary can then steal unencrypted received messages, private keys and group kety. The above detection method only lasts as long as the app itself is not compromised.
There are some concerns about group management in E2E protocol. You can read in this article, WhatsApp, Signal group chats not as secure as users might believe. Parent article has some more information about this problem
Technical talk from the Real World Crypto conference has also highlighted the problem with handling of group management in WhatsApp and Signal: On the end to end security of group chats | Paul Rösler (U. Bochum) | RWC 2018
You can also read their paper
WhatsAppp whitepaper has in-depth technical explanation of how key exchange for group chats is performed, pdf warning](https://www.whatsapp.com/security/WhatsApp-Security-Whitepaper.pdf).
I'm not sure about how other e2e protocols handles group management. Signal is the most popular one and widely tested.
