You have to check for all ways to access the device. For example:
- Is there some sort of serial interface that already has a console on it
- is there some serial interface that can be used to access the boot loader
- is there some serial interface that can be changed to have a console on it
- can the flash chip be accessed and changed directly (usually yes, you can only make it harder)
- are there any services listening on the network interface that can be used to change the filesystem
- are there any services listening on the network interface that have bugs which can be used to change the filesystem
apart from all that, it is always possible to directly change the (flash)memory contents, if they are not encrypted.
You can make this harder e.g. by drowning the flash chips in lots of resin (make sure they don't overheat) but that just makes it harder.
The most secure way is to use hardware that has support for encryption in the CPU/MCU and secure storage for the encryption key. This way, the flash storage is always encrypted and can't be changed. For even more security, the RAM contents also have to be encrypted.
This probably discourages most attackers. Still, modern game consoles use quite similar security measures and all of them got hacked eventually (due to bugs in the implementation/hardware).
The only thing you can do is decreasing the attack surface until it is not economically feasible to break the security. If redeveloping the device from scratch is cheaper than attacking yours, there is not much point in securing your device more.