There are lots of software for mobile operating systems providing end-to-end encryption. What is the best way to decide which one is trustworthy if their source is not revealed? Reverse engineering android
.apk files or trying to monitor network traffic?
Monitoring network traffic is a way to see how easy it might be to break the encryption head-on, but it will tell you nothing of how secure the app itself is, and only if they happen to choose a very weak standard would you find anything of interest. Rather, the greatest risk is an app that will give up the keys in a specific way (either locally or remotely) to allow eavesdropping. This is impossible to know unless you have the full source for the app, and everything else running privileged on the phone (which could snoop on the app). The team behind Signal (Open Whisper systems) publishes their source code via Github.
The best way to decide which one is trustworthy in my opinion is to revert to external professional audits. If the source is closed and additionally, there haven't been any audits, its really hard to prove the correct implementation of encryption, absence of bugs and backdoors and weak points in design.
If you want to audit the application yourself, reversing of the apk and traffic monitoring and analysis should be a part of the process.
See the EFF Secure Messaging Scorecard for more Information on audited communication applications: https://www.eff.org/secure-messaging-scorecard