So I have android lollipop installed, which should be encrypted automatically with some startup password.
My question is, what happens if the phone get stolen while its on, and the screen is locked? (If I had time I would obviously wipe it through Google.)
Is the file on my phone (nexus) encrypted while the phone is on?
Since applications can continue to execute in the background, the disk encryption key has to exist somewhere in memory.
There are differences among various Android versions in how the encryption key is derived from a passphrase or PIN, but all versions use the dm-crypt block-level disk encryption service of the Linux kernel. (Unlike iOS, Android doesn't use file-level encryption or multiple encryption key levels; on iOS, some keys are dropped as soon as the screen is locked, rendering some files inaccessible.)
This means that the files on disk are always encrypted, but the kernel will transparently decrypt them for any application that has sufficient file system privileges to read them.
Basically, the security of a device in such a state (locked screen, disk encryption key loaded in kernel memory) depends on the presence or absence of any security bugs in the Android lock screen or any installed application that can be remotely or locally exploited by an attacker; in that respect, there is no difference to an unencrypted device.
However, encryption protects against all attacks that require rebooting the device: As soon as the power is cycled, the encryption key is wiped from memory.
As commenters have noted, it should be added that an attacker with physical access to the device has more opportunities to gain access to the key material in memory than just exploiting kernel or application bugs: They could physically or logically extract the DRAM contents (which contain the Kernel memory and thus also the key) or gain access to debugging facilities of the system (assuming that they are not disabled).
