I'll dare it
Although objectionable if you are super pedantic, I am inclined to say "no", or "mostly no". It always depends on what you assume the attacker can do and will do, what's practical and what isn't. But when it comes to situations where one would say "yes", other problems are much more serious than someone unlocking your phone. Someone who can do this can also have you disappear entirely. Or, file a CLOUD warrant, for that matter.
Modern phones have specialized hardware which magically encrypts and decrypts your data, with the key never leaving the magic chip. That magic chip also has special provisions ("fuses") that physically destroy its ability to do so, given the right circumstances. Different manufacturers call their magic differently ("Secure Enclave", "Knox") but it is more or less all the same. Bottom line: Getting to the data is hard, very hard. Plus, you can always use Veracrypt in addition to the device's built-in security, which is more or less middle-finger-up towards anyone successfully unlocking the device and not knowing your decryption key (which isn't stored on the device).
What can someone with phsyical access to your device do?
Getting your "help"
Use a baseband exploit and gain root access, install spyware while you have the phone unlocked. Done, and thank you for your assistance.
Erm... alright, no physical access is necessary, so... I'm safe to say "no problem!" (in the context of the question).
The same goes for showing a fake error screen on a fraudulent web site, or making a phone call and telling you that our firewalls have detected a serious phone problem on your end which disturbs the internet in such a way that it requires me to reconfigure some servers at Apple to make it work again (this is what actual MS-PC scammers/extortioners claim, more or less verbatim, and people buy it!). So, to configure the Apple servers correctly, I need to know your password, and you need to follow a certain number of steps which I'll tell you. Physical access is not a prerequisite, stupidity is enough.
Steal the device, factory-reset it, and sell it on eBay.
There's not much of a defense against that. There's "find my device" and the like, and remote lock, and whatnot, but bleh. 99.999% certain you're not getting your phone back.
On the other hand side, in terms "Ow, my data!", this is a total non-issue.
Install a different firmware while you are getting your coffee
In theory, that's quite possible. In practice it's deep in the "yeah right, good luck" realm.
A factory reset (and firmware install) can usually be done without knowing the unlock password or having the correct finger attached to one's hand. Plugging in a cable and pressing the magic on/volume button sequence is enough. That's right.
There is however a small "but". First, doing so will erase all data. Second, doing so with a firmware that doesn't have what the phone considers a valid certificate issued by the phone's vendor kicks off the SE/Knox chip. Third, the process takes 15-20 minutes, so you need to have access to the device for quite some time. Do that, covertly, while hanging upside down from the ceiling as the target is getting a coffee? You should apply for the next Mission Impossible movie, you're certain to replace Tom Cruise. That's a more awesome stunt than the IMF break-in. And despite being an awesome stunt, it won't reveal the data!
Unlock the device bypassing biometrics
That may actually work, depending on how unaware or stupid the phone's owner is. If it happens, and if that is your only authentication, and you did not take other measures, then yes... you have lost. Seeing how you actually ask such a question, that's unlikely to be the case because apparently you have been thinking about the topic.
Facial recognition has been demonstrated to be rather not too safe, and the story with Samsung in-display scanners a year or two ago was a quite fun experience. In case you didn't know, there was a huge "security problem" with Samsung's in-display fingerprint scanners. Truth is, the scanners worked 100% correctly and did exactly what they should do, it was the users being too stupid. Users would put on a poorly manufactured, cheap protective gel screen protector from Mr.China on their $1000 phone, which necessarily had some recognizable, constant patterns. Thus, every time the user would lay a finger upon the scanner, the scanner recognized the finger, at different angles, and with different scale, pressure, ridge depth according to pressure, and the like... plus a recognizable pattern which was also always present.
So consequently, the AI learned -- of course, what else -- that this pattern is a valid part of the owner's fingerprint. Do it a few thousand times, and everybody can unlock your phone. No surprise there. That's stupidity though, not a defect.
Unlock the device by entering the PIN/Password
Well, hopefully, that is "Yeah, nice try", again it depends on how aware/unaware/stupid the user is. Your password/PIN isn't
0000 by any chance?
For a reasonable password, and with the "erase data after X failed logins" the practical risk is zero (yes it is not zero, but it is for all practical purposes it is). Oh darn it, I'm Sherlocked.
Replace the device
Replace the device with an identically looking one, and perform a relay attack, both with your fingerprint (... which they could get a million times easier, but not in a nearly as cool way) and the PIN and whatever. That, uh, is possible. Mission Impossible possible. Producing an identical phone with the same physical appearance and an identical-looking lock screen may be not that much trouble, but without already having had access to the device, getting the start screen and all to be exactly right is a real challenge. Oh darn it, I'm Sherlocked again, that one I didn't see coming.
Open the device and extract the key from the SE/Knox chip
Following that, decode the flash memory. Yes, this is possible. But it is, again deep in the "fucking heroic, Mission Impossible" realm. These chips are explicitly designed to make this a tough job. You will need to work with extreme care on a very small scale, read out a minimal electric charge, and even then there is a good chance you ruin it.
How do we know it's hard? Well because if it was actually doable by a reasonably educated and experienced lab assistant with decent tools, then the FBI wouldn't have made such a darn humiliating fuss about it (which Apple used very diligently for advertising) on that terror suspect phone a couple of years ago.
Realistically, it's mostly "no problem" for the "phone got stolen" issue, other than obviously you are lacking one phone and need to buy a new one.
Besides, the overwhelming majority of users stores their data on cloud services, both for backup and for sharing between devices, run by someone (possibly third parties, and likely affiliated with US-based firms) on servers in unknown locations, probably subject to regulations such as SCA and CLOUD, with unknown people having unknown levels of access. And, with unknown, if present, encryption.
Seeing how CLOUD has a "must" wording irrespective of physical location, the assumption "no encryption" is a very valid assumption. Otherwise there'd be no way for US companies that they could comply with the law.
Also, the fact that you can trivially, without hassle, share data between several devices kinda suggests that there can't be much of a secure encryption scheme in place, otherwise how would a different device that doesn't know the decryption key be able to use the data! You demonstrably don't need to transfer or know an encryption key or something, you just sign in (with OpenID if you like), and the data magically appears on your other phone (which now diligently encrypts the data in an unrecoverable way!).
I'd consider that much more worrysome, if the data on your phone is important.