1

For my personal research, I have to create a messaging app, but security is only important part of application, security from MITM (man in the middle attack), at device end, and at server level. (note: this will not end to end encryption, as I want to backup messages on server for cross-device access like telegram).

I created the following model:

  1. Server creates RSA key pair first time at installation and keeps private key in a file on server, and broadcast public key with its clients, (A, B)
  2. A&B also create RSA key pair at first-time login, and stores private key at device level and sends public key to server.
  3. A sending message to B:
    1. create a random salt_key
    2. encrypt message with this salt_key (AES)
    3. encrypt salt_key with server's public key (RSA)
    4. send RSA encrypted key, and AES encrypted message to server
    5. server can decrypt salt_key with RSA private key and then message
    6. server will do same as A did, but with B's public key and B will decrypt salt_key with private key then decrypt message.

Is this a secure way, or any another way to secure my messaging app?

  • 4
    See the signal protocol and its open-source implementation. – kelalaka Dec 27 '19 at 17:47
3

Ok, let's look at your proposal:

  • each of the participants (the server and clients) has a key pair, and the private key is only known by its owner: good
  • all messages are encrypted with a random symmetric key, and that key is encrypted with the public key of the next hop: not bad. Possible caveats: the choice of the random key (hell lies in details...) - be sure to choose a cryptographic secure random algorithm

Other points you should consider:

  • the server is highly sensitive because it knows, and keeps in clear text (or invertible encoding which is close where security is a concern) every message. It may be a requirement, but that means that you will have to prove that it cannot be easily compromised - the question here is who has either an admin account or physical access to the server or its storage
  • as you are speaking of devices, I assume that the secret key is just stored on the device. Here again nothing bad with that, but you should wonder who can access the devices
  • in your protocol, the messages are not signed. That means that they can only be read by their declared recipient, but you have no proof for the sender. It may meet your requirements, but is uncommon when security is important
  • the S/MIME standard is close to your proposal, except that the symmetric key can be encoded by multiple recipient public keys, and that this allows end to end encryption. Whether it is an option in your use case, only you can say...
| improve this answer | |
  • Hi, thanks for clearing my doubt. i know server is still a big risk but problem is backup, that's why i can not end 2 end encrypt, what if a user lost or break his device, and again sign-in to new device ? how he able to recover old messages or messages that not yet delivered to him during device change time. So looking best security practice. – Jaikey Sarraf Dec 31 '19 at 6:26
5
+50

This is actually just the opposite; it is a rather in-secure way of handling messages. Repeat after me: "My server should never, ever, be able to read the contents of a 'secure' message." Because you encrypt everything to the server who then re-encrypts & sends it on to someone else, if there is ever a compromise of the server then the entire system is compromised.

You really need to instead think of your messages as two separate parts, and to keep it conceptually simple we'll think of it as an old-fashioned snail-mail. You have an envelope and a letter. The envelope is readable by the mailman/post-office (your server), but they can't open it and read your letter, that's only for the ultimate recipient. In the scenario you describe, it is more like a postcard where the post office can read the message (you don't want that).

Instead, consider a scenario more like this.

You have a server that handles two functions: message delivery (+ storage while waiting for offline clients) and recipient lookups. You have clients that connect to the central server, their whole job is message encryption & decryption (and any local storage/filing of messages you want to include).

For security against the cases you listed, you need multiple layers of protection. All communications should be over a transport-encrypted link to prevent Man-in-the-Middle attacks (sounds like a perfect match for TLS). The server should generate an RSA key pair for itself, then as a recipient lookup function, it should publish the public component of this key in its own internal directory (prob with a reserved name that cannot ever collide with a user name). Each client should also generate an RSA-styled keypair for message protection (or import an existing key pair, if you want to join multiple clients to your account you'd import the same key pair onto each client); when this key is generated the public component should be sent to the server (along with a lookup value, think username or email address). The server would then add this key to the public directory. An important security feature is that the server should be able to translate lookup-values into public key IDs (and public key IDs into the full public key -- break that into 2 steps to avoid unnecessary data transmission, making things faster); but the server should never accept the public key ID/data and return the lookup value (with the encryption key a public user shouldn't be able to look up who the key belongs to).

Whenever you want to send a message, you put in the intended recipient(s) on the client-side. The client looks up each recipient to get their key ID. If the client already has the corresponding public keys in the local cache it uses them, elsewise it requests the full pub-key data from the server (by ID) and caches/uses that. At submission, the client makes multiple encryptions, one of the envelope (it encrypts the list of recipient key IDs using the server's public key) and another of the letter (the message itself is encrypted, but only to the recipients, not to the server), and finally a signature of each of those (so that the server can verify it is a legitimate message that should be delivered, and so the recipients can verify the message wasn't tampered with while in transit). The message is submitted to the server which decrypts the envelope data and puts copies of the message in delivery queues for each intended recipient (by public key ID, not by username/email).

Periodically your clients should query the server to see if there are any messages waiting on them. The query should be validated by the server by requiring the client to sign a random message with their private key (so you don't deliver the messages, even if they are encrypted, to the wrong person). Just have the server generate a random byte array of sufficient size to have high random-entropy, then ask the client to sign it with their private key (which is verifiable with the server's stored copy of the public key) before releasing messages to the requesting client. If you want the server to contain a backup of all messages, then have the server save the full message (envelope and letter) -- for security, the server still doesn't decrypt the "letters" but it can retransmit them down to any client that is checking in (and if the clients share the same key pair, then they will download the same message set, and can decrypt it on their end)

Once the client has the downloaded messages, it can locally decrypt using its private key and display the "letter" to the end-user. This guarantees that the message itself cannot be seen by the server, thus filling in the security hole that you had in the original design.

Basically, use your server like a PGP keyserver, MTA and storage point; and make your clients into the crypto-end-nodes. Your server can lookup public key data for any user, given that it is provided with the value (username/email/etc) to lookup --- but it will not provide the lookup value from the public key data (this prevents someone looking up who is communicating if they manage to compromise a message envelope). All of the message content is encrypted directly between clients (this prevents a server-side attack from compromising the message content, though it may allow the attacker to see who is communicating since the server has a database somewhere that maps key IDs to lookup values). The envelope is signed by the sender, this verifies to the server that the transmission is from a legitimate source (otherwise it should discard the message as an attempted attack). And the letter is signed by the sender, this lets the recipients verify that the message is really from the sender (prevents a MitM or server-side attack from sending messages to the "recipients" that aren't really from the purported sender).

Layering all that inside TLS for good measure (helps prevent MitM attacks) and you have a pretty durn secure system. You can ratchet it up by adding things like "password protect the private key data" (so a compromised client node doesn't immediately give stored message content away) or time-based-envelope/letter expiry (so that messages are transient and don't persist). On the server-side you could also do thinks like holding all your queues as encrypted data that only resides in memory (if someone steals the server and try to leave with it so they have time to brute-force your keys, the loss of power will also lose the messages sitting on the server, yielding a physical server attack less effective)

This isn't quite as strong as a full double-ratchet system, but it is nearly so. And it is fairly trivial to implement (RSA keys and PGP en/de-cryption are known processes that are rather easy to drop into a project).

| improve this answer | |
  • The OP specifically stated that end-to-end encryption is not the goal. As a result I suspect that your otherwise excellent answer largely misses the question here. – Conor Mancone Dec 30 '19 at 19:20
  • 1
    I agree that OP says they don't want end-to-end, but the reason is faulty. They want the server to have a backup of plain-text so they can put the messages on multiple devices. Instead they simply need the client to install the same key-pair on each of their multiple devices, and the server can backup the encrypted messages. Because while not wanting end-to-end, they do want actually secured against server-level attacks -- which basically requires end-to-end encryption (or, at least, never decrypting on server). Edited answer to better point out where that would come into play. – Ruscal Dec 30 '19 at 19:27
  • hi, @ruscal, thanks for answer, both answer cleared my some doubt, 1st answer by 'serge', ensured my approach is enough secure if server is not in goal and only i need to verify sender. I also found security trick from you that is " Instead they simply need the client to install the same key-pair on each of their multiple devices" which will ensure security even if server compromised. But you wrote this in comment not answer so accepting 1st answer but very thankful to you also voting up your answer which help users to know server is also very important part. – Jaikey Sarraf Dec 31 '19 at 6:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.