It depends. Your firmware may allow you to enroll certain signing keys for secure boot, and you could sign your existing OS plus the bootable material on the flash drive, and nothing else. However, you'd have to be able to remove the pre-set keys, which you may or may not be able to do, depending on the firmware.
Most firmware also allows you to set a supervisor password and prevent booting external drives without it. If you did that, then you wouldn't be able to boot your flash drive, but you'd be able to log in as the supervisor and modify the boot settings, which you'd presumably avoid unless it were the special drive.