I will first address the various attack vectors possible if the device is on or off and whether full disk encryption (FDE) is on or off. After that, I want to comment about the need for app-level encryption and the capabilities that it offers in your particular scenario.
Here, we assume the attacker has obtained control of the device. Furthermore, we assume that he or she is a well-educated hacker who has the time and resources to extract data out of the device. In real life, you may meet these people (but you may also meet those who are just interested in wiping the data and reselling the device).
Oftentimes, it comes down to the user’s habits that ultimately determine the security of the device. Let me mention a few of them (non-exhaustive).
(a) How does the user unlock his device? Pin code, swype pattern, picture, Touch ID, or none at all? Pin code is the strongest if there are sufficient digits. Swype patterns can be deduced by shining some light onto the phone and tracing out the oil stains. Picture and Touch ID are not fool proof either. See proof of concept videos on YouTube showing Touch ID being hacked.
(b) In addition, is the user conscious of his surroundings when unlocking his device? People may simply peek and memorize the patterns.
(c) How many seconds of standby before the phone automatically locks itself?
(d) Is the user running third party lock screen apps or any other types of apps which may introduce specific security vulnerabilities?
If the phone is unlocked, then the next thing stopping the attacker is any security authentication features built into your specific app (which I cannot comment on since I do not know your implementation). Just imagine you are an attacker now. How easy is it for you to launch up your app and see the data? What if the user recently switched apps? Does switching back to your app afterwards allow you the access the app or not (i.e. is the “login” persistent across app lifecycle states?)
Now, assuming the pin code is secure and the phone is locked at the time of theft, then a manual brute force attack (i.e. the attacker using his fingers to guess the password) is unlikely to work out. However, the attacker may try to do an automatic brute force attack using software tools. Just like ‘password recovery’ or ‘data recovery’ tools out there, you have to connect the phone to the computer and run the recovery tool. In this case, the phone is on. The success of running such forensic tools depends on the following:
(a) Is the device rooted or jail-broken into? Many recovery tools rely on this in order to get superuser rights and security privileges to crack the password or extract the data.
(b) Is developer mode on? Legitimate tools like Android Debug Bridge (adb) may be able to get data out of the phone if developer privileges are enabled. However, in later versions of Android, this should not succeed if the phone is plugged into a completely new PC since authorization must be given on the phone itself (an RSA fingerprint is registered). I am not sure about Apple or iOS.
(c) Is there a malicious or vulnerable third party app already installed in advance that the hacker can communicate with? For example, there may be some bad app that can remotely unlock the victim’s phone once the attacker gets hold of it.
P.S. If you can find a brand new device that can be hacked without the aforementioned conditions, do let Google or Apple know immediately, as this means their default OS implementation is insecure.
Notice that I did not mention FDE as it does not really help prevent the aforementioned attack vectors. If an FDE-enabled device is on at the time of theft, the OS is automatically triggered to do on-the-fly encryption and decryption of the disk sectors that are required by any app anyway. That’s how users can use their apps normally. Similarly, if the attacker’s tools are allowed to run on the phone, then FDE is not going to help, since the OS will cooperate and decrypt the information on-the-fly. In other literature, you may encounter the saying the FDE is mainly there to protect against offline attacks, and not online attacks.
Okay, let’s assume our phone has survived the attacker up to this point, due to our prudent user habits and the fact that we didn’t root and mess up the default security in our device. We have also enabled FDE, so the attacker cannot simply plug the phone in and read out all the data. Even if he transplants the hard disk from one phone to another, the other phone should not be able to read the data since the data is encrypted at rest.
The attacker is left with 2 main options to defeat FDE.
(a) Cold boot attack
(b) Direct memory access (DMA) attack
Now, cold boot attack takes advantage of the encryption key being in volatile memory (RAM). By cooling down the device, the electrical charge stays on for just a bit longer, long enough to plug into some other device/OS and read out the contents of RAM, thereby obtaining the key. In my opinion, this doesn’t seem as easy to do on a phone compared to a desktop with upgradeable memory, given that memory is built together with the motherboard. Also, this is quite impractical as the attacker has to train his skills on a specific hardware phone model.
Cold boot attacks might be mitigated if the phone is switched off at the time of the theft since the key should not be released to memory, however it does not apply to all systems! It ultimately boils down to how Android and Google have implemented their encryption schemes in their operating systems. At least for Android, I know the user has 30 tries to key in the right password upon turning on the phone before Android is booted. This seems to suggest that Android does not release the decryption key until the correct password is given. In a totally separate computing platform, Windows Bitlocker with TPM will release the decryption keys as long as it verifies that the boot partition hasn’t been tampered with; it has nothing to do with the user account credentials that one uses to log into Windows. In other words, shutting down the computer will not make a difference since the attacker can start it up again and the TPM will still release the keys as long as the integrity check passes. Of course, Bitlocker does come with two-factor authentication to counter this problem if you really want it.
In short, I just want to caution that while shutting down the device may seem a good practice, it is not a practical one in the mobile world, and it may or may not mitigate the cold boot attack depending on the details of the implementation.
The next attack used to defeat FDE is the DMA attack. Since Android phones do not come with DMA ports, I think this should not be possible. As for Apple mobile devices, thunderbolt may be exploitable, but I have not researched into this area to give a conclusive answer to this.
Now, let us move on to the real problem where FDE is concerned.
Security experts are happy that Android has finally caught up to Apple in providing default encryption with Lollipop, however, that’s only for new Lollipop devices. FDE is not enabled by default on pre-Lollipop Android devices (even if you upgraded from Kitkat to Lollipop, you still have to turn it on manually). That is a huge number of devices that would persist for the next few years!
Therefore, it may be wise to attempt to have an app-level encryption of our app data for a specific app, as you have alluded to. However, the devil lies in the implementation details. Many “secure” apps on the app store cannot really be trusted unless the source code is available for scrutiny.
One vulnerable implementation I can think of is to ship the app together with the encryption/decryption key. This sounds really convenient for the user – just fire up the app and everything works. No need to keep keying in passwords, since the key is stored somewhere. Okay, so the app data is encrypted and decrypted using the key is stored in plaintext. However, assuming that the device does not have FDE, the attacker can still obtain the key and use it the decrypt the app data (although it may slow him down if we do some obfuscation). This is definitely insecure.
One secure implementation I can refer you to is that of Truecrypt. It is an offline encryption software for computers that is open source. Although it has been a discontinued project as of 2014, experts believe it is still secure and some groups are trying to continue to maintain Truecrypt. It does not store the encryption key on the hard disk. Take a look at the details of its encryption scheme here http://andryou.com/truecrypt/docs/encryption-scheme.php . You can also check out the FAQ to find out more.
I believe you can adapt the scheme to your app. At the cost of convenience, make the user key in a password every time they start your app. The password will be input into a key generating function to generate a long AES key, and the key will persist in RAM until the app closes (like others have mentioned). It is never stored on the hard disk, In this case, it is still vulnerable to Cold Boot and DMA but it does not seem to be likely on mobiles at this point in time.
Another example I can point you to is Lastpass for mobile. It is a secure password manager that has extended its functionality to the mobile space. Maybe you can delve into how they implement their crypto schemes securely and how their app works (if I recall correctly, it also works offline).
Once again I like to stress that encryption does not defeat all attack vectors. If there is a malware running, then that malware could still get hold of the data while the user is using your app. Best weapon to fight this is user education.
Some final comments for now:
If a device is compromised, the user should do a remote wipe of the device. If FDE is enabled, the data is most likely unrecoverable since the key is lost after wiping.
In addition, the user should immediately change all of his passwords for active accounts logged into the compromised device.