For a trusted boot (or authenticated boot or secure boot or however you want to call it), you need a root of trust that's protected for integrity. It doesn't need to be a “super-secret” sector, it needs to be a “super-shielded” sector. ROM fits the bill.
In practice, device manufacturers like to put as little code as possible in the ROM, because they don't want to stop updating the code a few months before the device ships. So that code typically does nothing more than initialize a few core devices (e.g. put the MMU into shape), load the first few sectors from a storage medium, verify their cryptographic signature against a public key stored in ROM, and transfer execution to that code. If the signature doesn't match, the ROM code may allow the user to upload an alternate boot image (typically over USB).
That first non-ROM boot code is often considerably more complex than the ROM code: it usually needs to know how to manage memory, access various storage devices, perhaps read filesystems, have at least a rudimentary user interface and so on. So, in practice, all that code buggy, and that (or later in the boot chain) is how iPhones get jailbroken. Some manufacturers, such as Apple, are indeed deeply concerned with controlling the boot image; that doesn't mean they succeed against a determined community.
Furthermore, even if the signed boot image has held, the operating system itself is far too complex to withstand attacks for very long. Signing the whole OS is not practical as users of all but the most basic mobile phones and PDAs expect to be able to install their own applications (and OSes tend to have root holes). The OS can typically do pretty much what it wants, except that if it naively overwrites the boot image, the device won't boot. If you want to run a different operating system, you may be able to use the original OS as a jumping point from which you boot your real operating system.
Speed is another factor: verifying the authenticity of a 100kB bootloader is just a blip on a modern smartphone, but verifying the authenticity or the integrity of a multi-GB operating system is too slow to do at once. Incremental verification also slows down disk reads, and requires filesystem support that isn't widely available at this time.