Your general description is fairly standard end-to-end encryption. However, as in most cryptography systems, the main issue is key management, and that's where this can go wrong. For example, your scheme has no independent mechanism to verify the other party's public key. So consider the following scenario:
- Alice generates keypair and sends public key (A) to server
- Server secretly generates its own keypair (S)
- Bob requests A, but server provides S.
- Bob encrypts to S and sends to the server
- Server decrypts with S, then re-encrypts with A and sends to Alice.
Unless there is an independent way for Alice to confirm her public key to Bob, Bob has to trust that you're not doing that.
This is but one possible problem with the system; there are many other ways it can go wrong. Without a detailed description of the design, implementation, deployment, and maintenance, it's impossible to judge whether it's secure.
libsodium doesn't change anything here one way or another. It's a perfectly good cryptography library, but designing a secure messaging protocol is a lot more than just good cryptography. A great many systems fail in their key management. It is very easy to mess this up. I would not recommend designing a crypto system of any kind without engaging someone experienced in the field.