I spent some time seriously digging into the Truecrypt/Veracrypt source code and studying the issued audit, and I would bet one of my testicles that the container format is secure.
A simple TC reimplementation exists at https://github.com/bwalex/tc-play. It uses standard Linux kernel device mapper for mounting the TC volume. It reads the volume header, derives the encryption key from your password/keyfiles and seed stored in the volume header and uses the key and linux dev-mapper to mount the volume container. It uses standard symmetric key encryption implemented in the Linux kernel. It's the pretty same code as is used by LUKS - Linux crypto. So, if LUKS is safe than TrueCrypt is safe as well.
The only difference (as mentioned) is key derivation. However, it uses just standard crypto-primitives.
The only question is, whether the particular True/Vera-crypt implementation is safe. I would bet my second testicle that the tc-play is safe. It's short, and I was able to read it and understand it completely.
You can find even another reimplementation in linux cryptosetup. Many people have been thinking about TC security which could be nother proof.
If you look at the VeraCrypt sources, you realize it contains thousands of lines of code and 99% of the code is GUI. The possible backdoor could be hidden anywhere, and it would be complicated to find it.
If you compare VeraCrypt code and TrueCrypt code, you realize that VeraCrypt maintainer hasn't added anything interesting. He hasn't even fixed the known bugs discovered by the tc-play developer and noted in README. For instance:
"The keyfile pool is not XOR'ed with the passphrase but modulo-256 summed."
This is a documentation bug, which hasn't beend fixed in the VeraCrypt docs yet.
To implement new crypto-algoritm (which Vera does) is a simple task. Knowing how, you can put any new algo. within 1 hour of coding.