If I understand it correctly, this is a rough network diagram of your situation:
You're saying A and B want to talk, but they can't directly so they use Relay. You are thinking of having A set up an encrypted connection (TLS) to Relay and then using Relay as a proxy for connecting (over yet another TLS tunnel) to B.
This sounds complicated, but basically if you use an https proxy or any VPN server (say, mycompany.example) and connect to another https site (say, bank.example), you are already layering encryption on encryption. Let's see how the possible options work out (layering or not layering).
Option 1: encrypt A<->B but not A<->Relay nor B<->Relay
If you don't tunnel inside of the tunnel, you would be sending packets from A to Relay and from Relay to B unencrypted. But since they are destined for B, the traffic will be encrypted for B, nobody can read the connection contents (only A and B themselves). The only information you are leaking to anyone listening on the path between A<->Relay and Relay<->B is that A and B are communicating. The Relay will not be able to read the connection contents.
Option 2: encrypt A->Relay and A->B
If, like you're asking, there is a tunnel between A and Relay and between A and B, then people listening on A<->Relay can only see that A is communicating with Relay; and people listening on Relay<->B will again only see Relay communicating with B. The Relay will not be able to read the connection contents.
Option 3: encrypt A->Relay and B->Relay
Finally there is the option of encrypting the connection from A to Relay but not from A to B (and assuming B still sets up a (TLS) connection to Relay because it's behind NAT). In that case the traffic is transparent for Relay, but anyone observing will only see the connection from A to Relay and from B to Relay.
So it depends on whether it's an issue if people intercepting the lines can see the destination of the packets (namely A and B). Double encryption means double CPU power for A and B (only singular for Relay since it doesn't decrypt the contents, only the outer layer), but I'm not sure that makes a big difference or you would have mentioned it.
(Credits: the image was made with https://draw.io. The XML is in the source of this post.)