Can a hacker install a spying software or hack an unrooted Android device with physical access?
If your phone is powered on during access, and they have a way to get past your lock codes, I don't see why not.
In the end, phones nowadays are just computers, and if you give someone else physical access to your computer, it's not your computer anymore.
One example would be that the attacker disables the 3d party APK install protection and downloads his own RAT onto your device and installs it. Most bleeding edge phones even support OTG storage (You can plug in USB's with a USB(-C) -> Female USB cable), so that too could be a way to get malware on your device.