I'd like to ignore the comparison to WhatsApp because WhatsApp does not advertise itself as a "secure" messaging option. I'd like to instead focus on whether Telegram is secure.
Telegram's security is built around their home spun MTProto protocol. We all know that the first rule of Cryptography is Don't Roll Your Own Crypto. Especially if you aren't trained cryptographers. Which the Telegram people most certainly aren't.
Math Ph.Ds are not cryptographers. The protocol they invented is flawed. Here is a nice blog post explaining why. In addition to that, Telegram has issued a rather ridiculous challenge offering a reward to anyone who can break the protocol. Except that the terms they set makes even the most ridiculously weak protocol difficult to break. Moxie Marlinspike has a nice blog post explaining why the challenge is ridiculous.
So, no. Telegram is by no means secure. For commonly accepted definitions of secure, not the one Telegram made up.
I will compare Telegram and Whatsapp in 3 aspects: Storage of messages, encryption, and zero-day vurnerabitiles.
In fact I will be comparing 3 technologies: Telegram's regular chats (which I will refer to as "normal chats", or simply "chats"), Telegram's secret chats, and Whatsapp.
Let us assume that both Whatsapp and Telegram completely adhere to their privacy policies.
Whatsapp will not store your chats unless they are not yet delievered to the receipient.
An excerpt from WhatsApp's Legal notes:
Telegram stores all your ordinary chats.
Telegram does not store your secret chats. Even if they did it would be useless because it is end-to-end encrypted.
When it comes to storage, Telegram's regular chats are less secure than Whatsapp. However, when it comes to secret chats it is superiorly more secure than Whatsapp. Because the data remains encrypted until it reaches the receiving party's phone.
Diffie-Hellman End-to-end encryption is known to be practically unbreakable when used properly even if one does not trust the server passing on the messages. **
Therefore, secret chats are unbreakable assuming they were used properly and there isn't a serious flaw with the Telegram client software (read zero-day attacks section!).
"Properly" means authentication is achieved. In telegram this should be done by physically meeting with the other party and making sure the barcodes on the two devices are identical. Alternatively, the barcodes can be compared via a channel known to be secure
*Perfect forward secrecy can be emulated by constantly removing chats, re adding them, and re-comparing the keys. But that's hard in practice.
**It is still unclear to me what exactly is done after the keys are exchanged in secret chats, this is the key question to whether Telegram secret chats have proven security or not. Therefore, I opened a question regarding that.
3. Zero-day attacks / Security vulnerabilities
Therefore more eyes are inspecting the code, therefore the likelihood of a vulnerability is far lower. However, Telegram's server-side code is closed-source. But as stated earlier, one needs not to trust the server when one is using secret chats.
Whatsapp has a long history of security vulnerabilities which would have been detected early if it weren't the black box it currently is. (Links soon, you may Google "Whatsapp security" for now)
First place: Telegram's secret chats are the king of the ring (When properly used as explained above): Encryption is end-to-end, chats aren't stored and even if they are being stored data extraction is impossible thanks to end-to-end DH.
If you're worried about the fact that Whatsapp uses some undisclosed black magic which might have hidden vulnerabilities and you don't mind companies storing your chats, or believe that companies store your chats anyways, Telegram's regular chats come second.
Third place: Either Whatsapp or Telegram's regular chats, depending on who you picked second.
Other things to consider
*A rough English translation of the same link.
P.S: I don't have enough reputation to comment on Terry's answer, but I think that referencing the Unhandled Expression blog post is not a good idea because it is full of emotion and inaccurate info and it didn't objectively prove that the protocol is insecure. Please see the comments section of that blog post.
P.S 2: A non-standard crypto is not necesarilly less secure, and it is definitely more secure than an undisclosed crypto.
As the Telegram FAQ mentions, there is a 'secret chat' option that does not store chats on their servers.
As for the underlying question of, "does storing chats lower their security?" then that is something to consider. Chats being stored on the server does mean that copies can be made on the server for decryption later. This increases the exposure of the messages. Encrypting the messages means that there is a high cost to decrypt the messages, but there is still some exposure.
Taking this added exposure into account, the real question becomes (as it always does), "what are you protecting from?" If you are worried about secure communications in transit, then Telegram 'appears' to be more secure. If you're worried about secure communications at rest, then WhatsApp 'appears' to have a better model, except that none of it is encrypted.
The answer, then, is 'it depends on your focus', and encryption is better than non-encryption, and there is the Telegram's 'secure chat' option.
I doubt Telegram.
Telegram does not use established crypto libraries. Telegram is related to VK, Russia’s largest social network (VK = Facebook).
Telegram does not have a business model, but nothing is for free in this world. I do trust in Threema, I am not the only one.
The leading German consumer safety organization did an assessment on popular messengers and only Threema was rated as uncritical.
Here are two references: