First thing to do is define what you mean by "secure and authenticate". You need to put into words who the attackers are, what they can do, and what they should be prevented from doing.

On example of such an attack model is the following:

• Attackers are outsiders. They eavesdrop on the common channel (the radio waves). They want to obtain the confidential data that the group members exchange with each other. The group members trust each other.

If that is the attack model, then it suffices to come up with a shared secret K, that all group members but no attacker knows, and then encrypt packets with some symmetric encryption algorithm (say, AES in a suitable mode) using K as key. Initial distribution of K can be done by having each group member own a RSA key pair; one group member generates K randomly, encrypts it with the public key of each group member, and broadcasts that.

Now let's extend that attack model into another one:

• Attackers are outsiders. They eavesdrop on the common channel, but will also attempt to send fake messages, sometimes replaying previous messages, with or without alterations. Such fake/replayed message instances shall be reliably detected and dropped by group members.

Now you also need a MAC for each packet (so that the attackers can only replay previous packets, not create new ones or alter existing packets), and some sort of connection context so that replay attacks are detected. This can be done by embedding a sort of time stamp or counter in each packet: each participant remembers the counter values for the last packets received, rejecting packets which have already been seen or which have a "too old" counter value. Such mechanisms are often called window-based and there is some balance to strike between replay attack prevention, memory requirements (to remember past packets), and tolerance to packet loss due to the unreliable nature of the transport medium. Peruse DTLS as an illustration of how the SSL/TLS protocol can be adapted to packet-based unreliable mediums.

If the attacker is allowed to try to send fake messages, then he might also be able to advertise a fake RSA public key. Distribution of public keys is a matter of Public Key Infrastructure, with certificates being the usual solution.

Maybe we want more ? E.g., what about the following:

• Group members are attackers too. Each group member will try to eavesdrop on and/or interfere with one-to-one communications between other group members.

In that case, you will need pairwise key exchanges. For any pair of group members A and B, there must be a shared secret K_A,B that other group members and outsiders do not know. This can still be leveraged from pre-distributed RSA Key pairs: when A wants to talk to B and they don't already have a shared secret (or they forgot a previously shared secret, e.g. B rebooted his computer and that shared secret was in RAM only), then A generates a new random key K_A,B and encrypts it with B's public key.

There again, read up the complete TLS and DTLS standards: they are quite readable, and that will show you what you have to think about.

First thing to do is define what you mean by "secure and authenticate". You need to put into words who the attackers are, what they can do, and what they should be prevented from doing.

On example of such an attack model is the following:

• Attackers are outsiders. They eavesdrop on the common channel (the radio waves). They want to obtain the confidential data that the group members exchange with each other. The group members trust each other.

If that is the attack model, then it suffices to come up with a shared secret K, that all group members but no attacker knows, and then encrypt packets with some symmetric encryption algorithm (say, AES in a suitable mode) using K as key. Initial distribution of K can be done by having each group member own a RSA key pair; one group member generates K randomly, encrypts it with the public key of each group member, and broadcasts that.

Now let's extend that attack model into another one:

• Attackers are outsiders. They eavesdrop on the common channel, but will also attempt to send fake messages, sometimes replaying previous messages, with or without alterations. Such fake/replayed message instances shall be reliably detected and dropped by group members.

Now you also need a MAC for each packet (so that the attackers can only replay previous packets, not create new ones or alter existing packets), and some sort of connection context so that replay attacks are detected. This can be done by embedding a sort of time stamp or counter in each packet: each participant remembers the counter values for the last packets received, rejecting packets which have already been seen or which have a "too old" counter value. Such mechanisms are often called window-based and there is some balance to strike between replay attack prevention, memory requirements (to remember past packets), and tolerance to packet loss due to the unreliable nature of the transport medium. Peruse DTLS as an illustration of how the SSL/TLS protocol can be adapted to packet-based unreliable mediums.

If the attacker is allowed to try to send fake messages, then he might also be able to advertise a fake RSA public key. Distribution of public keys is a matter of Public Key Infrastructure, with certificates being the usual solution.

Maybe we want more ? E.g., what about the following:

• Group members are attackers too. Each group member will try to eavesdrop on and/or interfere with one-to-one communications between other group members.

In that case, you will need pairwise key exchanges. For any pair of group members A and B, there must be a shared secret K_A,B that other group members and outsiders do not know. This can still be leveraged from pre-distributed RSA Key pairs: when A wants to talk to B and they don't already have a shared secret (or they forgot a previously shared secret, e.g. B rebooted his computer and that shared secret was in RAM only), then A generates a new random key K_A,B and encrypts it with B's public key.

There again, read up the complete TLS and DTLS standards: they are quite readable, and that will show you what you have to think about.