Can malicious program open a file from mounted container?
If the volume is mounted, any process that has the rights to read the mounted filesystem will be able to open files in it exactly as if it were unencrypted. As soon as it is mounted, it is not encrypted at all from the perspective of the computer. This encryption can only provide data-at-rest security. That is, if the computer is ever stolen and someone gets their hands on the encrypted volume, they will not be able to decrypt it without knowing the original password you used.
Should I worry about data in memory when working with files?
The issue is not that the files are present in memory so much as the fact that the data is not being protected when the volume is mounted. The system will happily decrypt what you ask! When you try to access a file, whether it's present on a partition that is encrypted or not, the requesting program will tell the operating system kernel (the core of the operating system) what file it wants to access. The kernel will then perform all the steps necessary to read the file, including transparently decrypting it and handing the decrypted data to the requesting application.