TL;DR: No, Telegram is not secure.
I'd like to ignore the comparison to WhatsApp because WhatsApp does not advertise itself as a "secure" messaging option. I'd like to instead focus on whether Telegram is secure.
Telegram's security is built around their home spun MTProto protocol. We all know that the first rule of Cryptography is Don't Roll Your Own Crypto. Especially if you aren't trained cryptographers. Which the Telegram people most certainly aren't.
Math Ph.Ds are not cryptographers. The protocol they invented is flawed. Here is a nice blog post explaining why. In addition to that, Telegram has issued a rather ridiculous challenge offering a reward to anyone who can break the protocol. Except that the terms they set makes even the most ridiculously weak protocol difficult to break. Moxie Marlinspike has a nice blog post explaining why the challenge is ridiculous.
So, no. Telegram is by no means secure. For commonly accepted definitions of secure, not the one Telegram made up.
As of May 2016, this answer is out of date. Whatsapp rolled out E2E encryption.
I will compare Telegram and Whatsapp in 3 aspects: Storage of messages, encryption, and zero-day vurnerabitiles.
In fact I will be comparing 3 technologies: Telegram's regular chats (which I will refer to as "normal chats", or simply "chats"), Telegram's secret chats, and Whatsapp.
0. The fast answer
"end-to-end encrypted, needs more peer review" (Telegram)
Is, in my opinion, better than
"Not end-to-end encrypted, uses closed stuff which cannot be peer reviewed" (Whatsapp)
Opinion aside, here are the facts...
For now, let us assume that both Whatsapp and Telegram completely adhere to their privacy policies.
Whatsapp will not store your chats unless they are not yet delievered to the receipient.
An excerpt from WhatsApp's Legal notes:
Telegram stores all your ordinary chats.
Telegram does not store your secret chats. Even if they did it would probably* be useless because it is end-to-end encrypted.
*See 2. encryption.
When it comes to storage, Telegram's regular chats are less secure than Whatsapp. However, when it comes to Telegram's secret chats it is superiorly more secure than Whatsapp. Because the data remains encrypted until it reaches the receiving party's phone.
Diffie-Hellman End-to-end encryption is known to be practically unbreakable when used properly even if one does not trust the server passing on the messages.
Therefore, Telegram's secret chats are unbreakable assuming they were used properly and assuming there isn't a serious flaw with the Telegram client software or the crypto. However, as stated earlier, the crypto needs more peer review.
Whatsapp's protocol is undisclosed, we may never know how secure it is.
"Properly" means authentication is achieved. In telegram this should be done by physically meeting with the other party and making sure the barcodes on the two devices are identical. Alternatively, the barcodes can be compared via a channel known to be secure
*Perfect forward secrecy can be emulated by constantly removing chats, re adding them, and re-comparing the keys. But that's hard in practice.
3. Zero-day attacks / Security vulnerabilities
Therefore more eyes are inspecting the code, therefore the likelihood of a vulnerability is far lower. However, Telegram's server-side code is closed-source. But as stated earlier, one needs not to trust the server when one is using secret chats.
Whatsapp has a long history of security vulnerabilities which would have been detected early if it weren't the black box it currently is. (Links soon, you may Google "Whatsapp security" for now)
First place: Telegram's secret chats are the king of the ring (When properly used as explained above): Encryption is end-to-end, chats aren't stored, and even if they are being stored: data extraction is probably* impossible thanks to end-to-end DH.
*See 2. encryption.
If you're worried about the fact that Whatsapp uses some undisclosed black magic which might have hidden vulnerabilities which allow someone to snoop your conversations over the air, and you don't mind companies storing your chats, or believe that companies store your chats anyways, Telegram's regular chats come second. Because needs-more-peer-review is better than uses-unknown-stuff.
Third place: Either Whatsapp or Telegram's regular chats, depending on who you picked second.
Other things to consider
*A rough English translation of the same link.
P.S: I don't have enough reputation to comment on Terry's answer, but I think that referencing the Unhandled Expression blog post is not a good idea because it has many factual errors e.g.
As the Telegram FAQ mentions, there is a 'secret chat' option that does not store chats on their servers.
As for the underlying question of, "does storing chats lower their security?" then that is something to consider. Chats being stored on the server does mean that copies can be made on the server for decryption later. This increases the exposure of the messages. Encrypting the messages means that there is a high cost to decrypt the messages, but there is still some exposure.
Taking this added exposure into account, the real question becomes (as it always does), "what are you protecting from?" If you are worried about secure communications in transit, then Telegram 'appears' to be more secure. If you're worried about secure communications at rest, then WhatsApp 'appears' to have a better model, except that none of it is encrypted.
The answer, then, is 'it depends on your focus', and encryption is better than non-encryption, and there is the Telegram's 'secure chat' option.
New research shows deep problems with the crypto: https://medium.com/@thegrugq/operational-telegram-cbbaadb9013a#.gb7od1j6i
Besides the protocol issues, the app itself is not very secure. In February 2015, Zimperium published a detailed analysis of Telegram's local vulnerability, allowing the attacker to get full access to plain text messages.
Basically, even if the protocol was secure, the application itself isn't, becoming the weak link in secure communication.
According to Zimperium, the Telegram team has never responded to their vulnerability notification. It tells me something about their attitude to security in general, and goes in line, for example, with how they implement "secure chats": no desktop support, graphical-only fingerprint of the key, no possibility to simply enter the key.
EFF compares all messenger apps and publishes the results in Secure Messaging Scorecard link.
note:EFF compares Telegram in secret chats mode with WhatsApp
EFF criteria are:
finally,the result is that Telegram is more secure than Whatsapp
protected by Community♦ May 8 at 10:16
Thank you for your interest in this question.
Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).
Would you like to answer one of these unanswered questions instead?