It depends on the target machine. Some systems are configured to "wake up" when a special "power up" key is pressed on the keyboard; a USB device could pose as a keyboard and emulate such a key press. Also, some machines are seemingly off but are just hibernating, which means that they are still on, or ready to go live quickly.
In any case, as @tylerl says, no data will be read from the hard disk until the disk is spinning. So such an attack device must be able to make the machine live again.
Once the machine is convinced to boot up, the attacker's interest is to boot it up on an operating system that the attacker controls. The USB device can also pose as a simple Flash drive, and hope for the machine to be configured to boot on any USB Flash drive which would be plugged in the machine at that point. Most machines are not configured that way, though. A sneakier way would be to make the USB device "look like" a DVD reader: there are machines which won't boot from any Flash drive, but will consider CD/DVD readers specially and try to boot from that.
There are other options, though. For instance, the attacker could let the machine boot on its normal OS, then hack into it by claiming the USB device to be some sort of USB-to-FireWire converter: the target machine may have a ready driver for that, or even be so good as to download one automatically in that case, and that driver will enable DMA from the device. See this answer for details. This allows the attacker to hijack the machine and its OS, at which point harddisk-reading will be easy.
More esoteric options imply exploiting security holes in the USB controller driver (a part of the OS). It has been done for some PlayStation 3 consoles.
A simpler method, though, might be to open the computer case and grab the disk altogether. An attacker who can plug arbitrary devices in USB port is an attacker who has physical access, and he may have a screwdriver with him. In some specific situations (e.g. one of these "print stations" where you can print your own photos), a USB-only attack could be more practical and discreet for the attacker, but in most other contexts physical access equates to total loss of confidentiality, so this reading-through-USB thing might not be as devastating as it initially seems to be.