This is actually just the opposite; it is a rather in-secure way of handling messages. Repeat after me: "My server should never, ever, be able to read the contents of a 'secure' message." Because you encrypt everything to the server who then re-encrypts & sends it on to someone else, if there is ever a compromise of the server then the entire system is compromised.
You really need to instead think of your messages as two separate parts, and to keep it conceptually simple we'll think of it as an old-fashioned snail-mail. You have an envelope and a letter. The envelope is readable by the mailman/post-office (your server), but they can't open it and read your letter, that's only for the ultimate recipient. In the scenario you describe, it is more like a postcard where the post office can read the message (you don't want that).
Instead, consider a scenario more like this.
You have a server that handles two functions: message delivery (+ storage while waiting for offline clients) and recipient lookups. You have clients that connect to the central server, their whole job is message encryption & decryption (and any local storage/filing of messages you want to include).
For security against the cases you listed, you need multiple layers of protection. All communications should be over a transport-encrypted link to prevent Man-in-the-Middle attacks (sounds like a perfect match for TLS). The server should generate an RSA key pair for itself, then as a recipient lookup function, it should publish the public component of this key in its own internal directory (prob with a reserved name that cannot ever collide with a user name). Each client should also generate an RSA-styled keypair for message protection (or import an existing key pair, if you want to join multiple clients to your account you'd import the same key pair onto each client); when this key is generated the public component should be sent to the server (along with a lookup value, think username or email address). The server would then add this key to the public directory. An important security feature is that the server should be able to translate lookup-values into public key IDs (and public key IDs into the full public key -- break that into 2 steps to avoid unnecessary data transmission, making things faster); but the server should never accept the public key ID/data and return the lookup value (with the encryption key a public user shouldn't be able to look up who the key belongs to).
Whenever you want to send a message, you put in the intended recipient(s) on the client-side. The client looks up each recipient to get their key ID. If the client already has the corresponding public keys in the local cache it uses them, elsewise it requests the full pub-key data from the server (by ID) and caches/uses that. At submission, the client makes multiple encryptions, one of the envelope (it encrypts the list of recipient key IDs using the server's public key) and another of the letter (the message itself is encrypted, but only to the recipients, not to the server), and finally a signature of each of those (so that the server can verify it is a legitimate message that should be delivered, and so the recipients can verify the message wasn't tampered with while in transit). The message is submitted to the server which decrypts the envelope data and puts copies of the message in delivery queues for each intended recipient (by public key ID, not by username/email).
Periodically your clients should query the server to see if there are any messages waiting on them. The query should be validated by the server by requiring the client to sign a random message with their private key (so you don't deliver the messages, even if they are encrypted, to the wrong person). Just have the server generate a random byte array of sufficient size to have high random-entropy, then ask the client to sign it with their private key (which is verifiable with the server's stored copy of the public key) before releasing messages to the requesting client. If you want the server to contain a backup of all messages, then have the server save the full message (envelope and letter) -- for security, the server still doesn't decrypt the "letters" but it can retransmit them down to any client that is checking in (and if the clients share the same key pair, then they will download the same message set, and can decrypt it on their end)
Once the client has the downloaded messages, it can locally decrypt using its private key and display the "letter" to the end-user. This guarantees that the message itself cannot be seen by the server, thus filling in the security hole that you had in the original design.
Basically, use your server like a PGP keyserver, MTA and storage point; and make your clients into the crypto-end-nodes. Your server can lookup public key data for any user, given that it is provided with the value (username/email/etc) to lookup --- but it will not provide the lookup value from the public key data (this prevents someone looking up who is communicating if they manage to compromise a message envelope). All of the message content is encrypted directly between clients (this prevents a server-side attack from compromising the message content, though it may allow the attacker to see who is communicating since the server has a database somewhere that maps key IDs to lookup values). The envelope is signed by the sender, this verifies to the server that the transmission is from a legitimate source (otherwise it should discard the message as an attempted attack). And the letter is signed by the sender, this lets the recipients verify that the message is really from the sender (prevents a MitM or server-side attack from sending messages to the "recipients" that aren't really from the purported sender).
Layering all that inside TLS for good measure (helps prevent MitM attacks) and you have a pretty durn secure system. You can ratchet it up by adding things like "password protect the private key data" (so a compromised client node doesn't immediately give stored message content away) or time-based-envelope/letter expiry (so that messages are transient and don't persist). On the server-side you could also do thinks like holding all your queues as encrypted data that only resides in memory (if someone steals the server and try to leave with it so they have time to brute-force your keys, the loss of power will also lose the messages sitting on the server, yielding a physical server attack less effective)
This isn't quite as strong as a full double-ratchet system, but it is nearly so. And it is fairly trivial to implement (RSA keys and PGP en/de-cryption are known processes that are rather easy to drop into a project).
