How to have client encryption without server encryption key

Question

I want to build a service where people can message each other via their mobile device so that they encrypt all of their information on their device and not on my server. Basically I want to two people to securely message each other and have confidence that my servers nor anyone else can have any knowledge or access to their data without their permissions. I'm especially wary of preventing mitm attacks.

Currently all my messaging runs through my rabbitmq servers in the US, but I want the data messages going through rabbitmq to be encrypted before getting to my mq servers. I understand that this is what SSL basically is, but is it really the best option out there especially if the certificate servers are US based? Is it possible to have dynamically generated client keys only stored on mobile devices that can't be cracked by me? What I'm afraid is for the government asking for my encryption keys to decrypt my clients traffic in case of an impending lawsuit.

Answer 1

First: The very first rule of cryptography: Don't roll your own crypto!

Second: Why do you want to setup this service? Signal (+ OTR or Axolotl) does all this already for: It's open-source, allows peer verification, offers end-to-end encryption, offers client-side encrypted storage and is trusted by all well-known cryptographers.
Disclosure: I do use Signal and I'm convinced of its quality, I'm not working for OpenWhisperSystems or otherwise affiliated.

---

But now, let's head over to actually answer your questions:

I understand that this is what SSL basically is, but is it really the best option out there especially if the certificate servers are US based?

SSL (or more modernly called TLS) usually provides transport security between the end-user and your servers. It does not by itself do anything beyond that. You could let the clients establish TLS sessions between each other (resulting in good end-to-end encryption) however TLS is more designed to be a (somewhat) real-time security protocol. Concerning the certificates, if you do use something else than the RSA key exchange, a leak of the private key won't allow TLS data decryption (if you deleted the ephemeral (EC)DHE keys) but would allow man in the middle attacks (this assumes you use anonymous / ephemeral (EC)DHE). Adapting a TOFU-style approach or another key-update mechanism could make the connection resistant to man in the middle attacks after the initial connection.

Is it possible to have dynamically generated client keys only stored on mobile devices that can't be cracked by me?

Absolutely. OTR already includes such a functionality. Basically you want to end up as a relay for the messages the clients transmit (much like the standard internet infrastructure) while they securely negotiate a shared secret via your servers. You can verify your peer than by comparing the hashes of the shared Diffie-Hellman secret (or the public key of the other party used to authenticate Diffie-Hellman public keys) out-of-band.

Answer 2

It sounds like you could adopt a schema similar to the one that ProtonMail uses for secure end-to-end encrypted email.

ProtonMail claims to have zero access to users' information. Each user generates a key pair at their own device, and ProtonMail has every user's public key. Messages are encrypted client-side, and only the encrypted messages are sent through ProtonMail's servers. For more info, see https://protonmail.com/security-details.

Answer 3

Though some on this thread say to not roll your own crypto system, I will argue otherwise that it is necessary to facilitate true learning. That being said, it's better to use a hardened solution vetted by other experts as mentioned by the people on this forum.

What you are looking for is client side encryption using asymmetric key encryption. As the scheme goes, you exchange public keys to whomever you want to talk to. A good algorithm to use is RSA public key. With RSA public keys, you can do things like... digital signatures to authenticate the person who truely has private key pair with the public key you have.

What you can do is have your app generate a key pair client side, and give the public key to your server. When someone else wants to talk to your client, your server will hand over that public key to that client and now you can exchange encrypted messages through RSA.

I've glossed over some details but that is how you would build a MITM resistant protocol. If theres anything you should gain from this,...

The RSA public key should only be used to encrypt, while the private key is used only to decrypt. Never hand out the private key to anyone, including your server. They say RSA 2048 is secure enough, but I normally do RSA 4096 just because.