Given that you've expressed that the cert alone is a sufficient identifier, I'll assume that you're asking "given that someone has seen a certificate go over the wire, can they brute force their way in as that user later?". (Vs "can someone wardial getting into account X by using all the certs they exfiltrated from some computers?" - which is an easy "yup")
The answer is: Of course. But anyone who would try to do so is foolish.
Given a certificate what a brute forcer is trying to do is determine the private key that goes with the public key embedded in the certificate. Doing a TLS brute force attack would be equivalently "try 000000...00000, 0000...000001, 000000...000002", which just comes down to signature forgery. But, since the attacker could themselves just compute the sign+verify operations they can do an offline attack; so by the time they get around to talking to your service they should have already determined they have the right answer (since they are just going to run the same formula you are, modulo whatever access rights you check for based on certificate properties).
They could also just start jamming self-signed certificates (or 3rd party CA signed certificates) at your service to see if you've done an overly simplistic authentication model (such as reading a value out of the subject without doing CA pinning); but that's not really "brute force" so much as "breaking your auth model".