I'm developing a public VoIP relay server, that will help bypass internet firewalls by allowing two clients to connect with it and then it will relay data between the two. Both clients will connect with an API endpoint to authenticate first and request a VoIP session, encryption keys etc. The sole job of the VoIP server is to relay data. I was thinking of providing a 64-Bit random number (ID) using a CNG to both clients along with the encryption keys - from the API endpoint. The way it will work is that when the clients connect with VoIP server they will identify themselves by just providing the ID in plaintext (no SSL). The relay server will look at the IDs (and whether they are valid, not used already etc) and pair the two connections internally and reply with a Confirmation message to both clients that the link is active. Then clients can use the encryption keys (AES) to start sending encrypted data to each other. The benefit I see of this approach is that that I don't have to write any encryption code for the VoIP relay server and will simplify the development of VoIP by a great margin! Do you guys think this is a secure setup? Is this prone to Man-in-the-middle attacks?
Some Clarification: Initially both clients will connect with the API EndPoint(RestFul Service) over a secure connection (SSL) to authenticate themselves, request a VoIP call, which will create two unique IDs (the pair would be sent to the VoIP Relay Server as well) and AES Encryption keys (only sent to the clients). The idea is that VoIP server does Not need to know anything about the connection so why waste development time and effort with setting up SSL. It will only pair two connections to each other, so just verify the IDs that each incoming connection is providing and then pair the two connections together (i.e. forward data from A->B and B->A). Clients will use AES encryption on their ends to encrypt data they are sending to the relay server but won't be TLS/SSL protocol. Regarding IDs, both clients will get a unique ID but the relay server would have both IDs, so it can pair them.