There are many types of CFI. However, CFI that supports both forward and backward edge protection, is deterministic, and is also fine-grained is very rare to come by (the only concrete example I know of is the commercial version of PaX RAP). Most CFI is coarse-grained, which means that a function can return not to only that one function, but to a number of functions with the same function signature (that is, the same return type and exact same arguments). Forward edges come from jumps and calls, whereas backward edges come from returns. The vast majority of CFI only supports one or the other (such as Microsoft's CFG or Clang's CFI), vastly limiting its capabilities.
Another issue with CFI is that it is often probabilistic, using some secret value. Any infoleaks that expose that secret value could be used to break the CFI. Deterministic CFI does not suffer from this, but most deterministic CFI is very coarse-grained. In summary:
Most CFI is not both forward and backward edge, severely limiting its capabilities.
Most CFI is probabilistic, making it vulnerable to infoleaks that can break it.
Most CFI is coarse-grained, letting you return to groups of functions with the same signature.
You can learn a lot more about actual CFI implementations by checking out kCFI and PaX RAP, the only two which I know of that provide full forward and backward edge protection. Unfortunately, since CFI is such a complex topic and there's no one implementation or even technique for CFI, there's no way to answer exactly how an attacker can bypass it without specifying the implementation.