We come across Android applications which use the custom encryption. When we proxy the Android phone or emulator, the proxy captures traffic which is unreadable.
Is there a known technique to read this traffic or reverse engineer the encryption?
It depends upon the encryption. However, if they are using modern secure encryption, then there will be no way to read this traffic from the proxy.
Your best bet will probably be to instrument or monitor the application so you can see the data inside the app before it gets encrypted. Alternatively, it may be possible to modify the application to turn off the encryption, or to make it less secure (e.g., if it is using SSL, modify it so it doesn't check the server's cert or so that it accepts your cert as a valid CA, and then mount a man-in-the-middle attack).
All of this is likely to involve some amount of reverse-engineering the code of the Android application, which may take some work if the app has been obfuscated.