I have been wondering for a while, since (for example on WindowsOS) logs are stored of user activity on the computer and can be retrieved later on by forensic experts easily. Can a user use an external hard drive or a USB storage device to install and run applications on it, which will not be technically stored on the computer's internal hard drive? Will the computer log the activity running on that external USB/HD?
The operating system of the computer (Windows, Linux, OS X, etc.) keeps its own logs of activity, and those logs are independent of the activity being performed. So whether you run a program off a local hard drive or an external USB drive, the OS of that computer will still log it in the same log it always uses. In Windows, these are kept in the Application Event log, and are visible to administrators of that machine. (If you're interested, you can go into Control Panel / Administrative Tools / Event Viewer and see them for yourself.)
A good security practice for a system administrator is to immediately forward copies of security events from the computer's log to a central collection server for backup purposes. That way, even if a hacker manages to get onto the machine and destroy the logs he or she finds there, the hacker won't be able to get to the copies that have already been stored elsewhere. The forensic investigator will then have information to help trace what's happened.
I'm going to piggyback on John Deters's answer. All major operating systems (Windows, Linux, OSX) keep logs of user activity for varying purposes, and they are stored in various locations.
With that in mind, if you don't want to have logs on your activity, check out an amnesic operating system like Tails. Tails is run as a live CD, only ever storing information in RAM which is erased on shutdown. This leaves little to no information for forensic experts to recover (unless they did something like a cold boot attack: https://en.wikipedia.org/wiki/Cold_boot_attack but this is largely impractical, especially if you have decent physical security).
Finally, remember that no operating system that communicates on the internet can be truly amnesic, because at some point it has to send data (encrypted or otherwise) through routers, servers, etc. Tails mitigates some of this by using Tor by default, but your encrypted traffic is still visible. Not to mention that the NSA can compromise tor traffic by controlling exit nodes.