In principle, you could modify the SoC (System on Chip)'s firmware (e.g., the simple bootloader that initially tells the chip to read the SD card to get the real firmware image to boot up) by uploading new malicious firmware which would persist after replacing the SD card.
However, I am not sure if the Raspberry Pi's SoC firmware is (easily) modifiable. I believe it is pre-programmed from factory and may not be rewritable; though a sophisticated attacker could always replace the chip with one maliciously factory-programmed.
There is firmware on the SOC that you will not be able to touch. It is pre-programed from the factory. And yes it is required to boot, and this is why the first partition has to be FAT because this is how the firmware is designed. You only need a FAT partition big enough to hold a bootloader of some type.
Also from http://elinux.org/RPi_Software:
The boards do not include NAND or NOR storage - everything is on the SD card, which has a FAT32 partition with GPU firmware and a kernel image, and an EXT2 partition with the rootfs.
We're not currently using a bootloader - we actually boot via the GPU, which
contains a proprietary RISC core (wacky architecture). The GPU mounts the SD card, loads GPU firmware and brings up display/video/3d, loads a kernel image, resets the SD card host and starts the ARM.
You could replace the kernel image with a bootloader image, and that would work fine.
Implying that there is some simple code in the SoC that on boot mounts the SD card and reads the firmware, that in principle, could be replaced with malicious code. Though this may require physically replacing the chip with something created in a chip factory that initially loaded malicious code.