This depends on the level of hardening. Having physical access to a Linux server generally means one is able to reboot it.
Next, when the Grub screen appears select "Advanced options". At this point you can edit the commands by pressing "e".
Find the line that starts with "Linux" and change the read only mode to read write mode. At the end of the line add
By pressing F10, you should be in a terminal. At this point you only need to mount the root file system in read/write mode:
mount -n -o remount,rw /
passwd and enter a new root password, reboot the machine and use the new root password for full access.
I would like to know if, by default (after a fresh installation), it
is possible to plug a USB into the computer and steal data
Yes, if the server has USB ports, it certainly is possible to plugin a USB device. However, stealing data requires access to at least the terminal with root privileges, which can be obtained as described above.