I can only control the access to the socket at user level (or user group)
Unfortunately, if there's no parent-child or sibling relationship between the processes, this will always be the case. The security model of all major operating systems, both Windows and Unix-like, is primarily user-based. Even when you can identify the specific process and binary image on the other end of an IPC channel, you can't tell whether or not some other process is looking into its memory or even outright puppeting it using debugger APIs or similar. You can't even necessarily tell if its the program you think it is (e.g. on Linux I could create a process /bin/AndreaClient that connects to your IPC but does what I want, and then delete the file and replace it with your official AndreaClient). It is possible to exclude some processes that have lower-than-full-user privileges (that is, processes which are less than "full trust" and run in a sandbox) easily enough - just don't make your IPC endpoint visible to the sandbox - but you can't prevent any process running with at least as much privilege as your usual client from connecting.
If the two processes are parent/child or similar, you can perhaps set up the connection between them without ever exposing it to other processes, by using
anonymous pipes to ensure the connection is already open before spawning the new process (and passing the open file descriptor directly to the new process). However, even there, you can't prevent the case that some other process (a debugger with the necessary privileges) attaches to the running process after the fact, and indeed you'd need to be careful how you launch the child process (open file, verify its correctness using e.g. cryptographic hash or signature verification,
execveat the still-open file descriptor) if there's any risk that another process would overwrite the child process binary.
Note that adding encryption wouldn't even help, here, because where are you going to store the key? Anywhere you could put the key - in a separate file, embedded in the binary (please don't), in an environment variable, etc. - it will be at minimum also visible to the superuser and any debuggers of either process, probably visible to all processes running as the same user, and possibly visible to all users.
With that said... why does it matter? If you don't trust the code running as the same user (or group), or the superuser, why are you trying to put any secrets on the machine at all? It's compromised! The attacker already has control, it's game over, man.
It's perfectly reasonable to block access to limited-privilege peer users, and to sandboxed processes, by putting your local (unix) socket / named pipe / memory-mapped file / ALPC endpoint, etc. out of reach of such untrusted entities (protecting it using file permissions / ACLs / mandatory integrity controls). Approximately every form of IPC (with the notable exception of loopback network sockets) support such access controls.
As for which form to use, that depends a lot on your use case. Local (Unix) sockets (which also work on Win10 and later) have the advantage of being standardized and cross-platform, and have some useful features (e.g. passing a file descriptor across them in a useful way), but they are also a fairly low-level communication tool; you'll get a binary stream you can read and write to, and need to design the protocol and implement the server and client on top of it. You can reuse existing protocols and libraries for them, of course, provided that the library accepts an open socket fd (or can open a local path instead of a network address).
Alternatively, higher-level communication channels may be available. Some common options (at various levels of abstraction) are Remote Procedure Calls (such as MSRPC which is technically cross-platform but de-facto Windows only, gRPC which is cross-platform, etc.), System V message queues (implemented on most *nix systems), D-bus (present on most Linux desktops and some other systems), Grand Central Dispatch (from Apple but technically cross-platform), and many others. Unless you're building for a very low-resource embedded platform, you probably have options that might be more convenient than local sockets. On the other hand, sometimes simplicity is desired; possibly all you need is a single unidirectional pipe (FIFO). In any case, all of these mechanisms are going to primarily be secured by the user-based security model of the underlying OS.