Is it conceptually possible to allow in the server a specific self signed client certificate for mutual TLS?
Yes. A self-signed certificate is nothing special. Using the trust chain against a trusted root CA is not the only way a certificate can be verified, but one can for example simply explicitly trust the given certificate or the public key inside it.
Note that a self-signed certificate still need to be verified against the expected value. Blindly trusting arbitrary self-signed certificates (as done in many examples) is insecure. How such validation needs to be implemented is different between programming languages and frameworks.
If possible but not recommended. Why?
Certificates issued by an already trusted CA can rely on this CA as the trust anchor. For self-signed certificates there is no such pre-existing trust anchor. In this case the certificate, its fingerprint or similar must be distributed using some existing trusted mechanism to all the peers which should trust the certificate. And it must be removed from all these clients if it needs to be revoked. This is doable with a few peers but does not scale to many.
I have a client to who I have to provide a server that does mutual TLS auth. But they say they wont sign our server certificate nor will they let us sign their client certificate. How should I approach this?
As I said, self-signing certificates are perfectly acceptable to use if the trust relationship was distributed over a trusted medium. It simply does not scale well with many parties and it might be a problem to properly verify the certificate in the applications. But, if this is not a problem in your specific case, then self-signed certificates can be used.