I need some starter advice here. Never done Java reverse engineering before, but I needed to somehow read the memory of a running desktop java application. The process communicates with an external server on an encrypted channel and fetches some files. The files as far as I know are Java bytecode meant to extend the functionality of the current application. I would like to know what are in those files, so I could later try to reverse bytecode to program code to get a closer look how it is to change the behavior of the application. Can you suggest a suitable debugging tool or tell me a bit about this methodology?
I usually use the Krakatau (dis)assember https://github.com/Storyyeller/Krakatau for examining and tweaking Java bytecode files.
The only way to get code into a JVM is to implement a class loader. The bytecode that goes into the JVM is, of course, in plaintext. So, disassemble the local classes first, locate the ClassLoader and hook the ClassLoader to dump the bytecode that would be loaded to a file (and possibly, load replacement code if you want).
Good luck, and don't do this in USA, as it is probably a DMCA violation.
Can you break the encrypted channel with a proxy (if it's HTTPS being use for encryption for example?) and install the proxy's SSL certs into your local cert store so that the process trusts the proxy? Then you can capture the data from the packets (i.e. the byte code) and go from there? This doesn't quite give you the image of what is actually running following a download but it may be enough for your purposes. If you can't do this (because it's not a proxyable protocol for example, or the process doesn't use the local cert store) then it depends on your local setup in terms of OS and level of privilege etc.