Attacker has only access to USB port and can either attach a device or pen-drive to a computer, that can be on or off. Imagine a scenario where non-reliable people have access to offices, either because they clean after-hours (off scenario) or because they sit near the running computer (on scenario) but only that, what is the risk to this computer? Could you connect a pen-drive to it when the computer is turned off and run some malicious software on it? If the computer is running, can you connect a cable to this USB port and extract some information?
If a USB port was available, any attack could potentially trick the computer into believing a small plug was a keyboard, making it possible to save and run scripts on the computer.
They could also run a program on your computer by using the Autorun feature on your computer, although that feature could also be disabled.
Practically anything can be done, but it's dependent on whether or not the computer will allow it to be done.
There was a great talk about bypassing endpoint security with USB stick appearing to be an authorized device on DEFCON two years ago: here.
There are many more, one being the issue of a USB multiplexer which can point to different things when it recognizes the USB device being plugged in (very interesting talk!) and another about fingerprinting USB stack.
From a powered-on, locked computer you may be able to extract network information by routing it via a USB network card becoming the default gateway. More complex attacks become possible by leaving behind keyloggers, or time-activated devices acting as disks upon reboot or keyboards after user logon. Embedded wireless can be used to exfiltrate information, possibly in conjunction with already installed malware (e.g. on air-gapped devices). The alleged NSA catalogue provides some examples.
Acting as a keyboard, the device sends the necessary button presses to bring up the boot menu and boots a minimal Linux system from the hidden thumb drive. The Linux system then infects the bootloader of the computer’s hard disk drive, essentially acting like a boot virus, he said.
Another proof-of-concept attack [...] involves reprogramming a USB drive to act as a fast Gigabit network card. [...] OSes prefer a wired network controller over a wireless one and a Gigabit ethernet controller over a slower one. This means the OS will use the new spoofed Gigabit controller as the default network card.
Note you can also insert a USB device inside any other USB device (e.g. a keyboard) so the rule of thumb becomes "If someone can touch your computer, it's not your computer anymore"