It seems somewhat unsafe to move the private portion of one's key around to multiple machines...but it also seems odd to have multiple identities floating around.
The private key is owned by a human, not by a machine -- indeed, the normal case is that you (as a user) have one key pair, and use the private key on all machines where you need to.
Having a key pair per machine would not work well; this would mean that when people send you an encrypted email, they will use one of your public keys, and you will be able to read the email only on the specific machine which contains the corresponding private key. This is hardly convenient.
We can say that as long as you want to read the same emails on several machines, then these machines must contain private keys which are functionally equivalent. Making these private keys distinct from each other is then a useless complication; just accept that you really want to have your private keys in several machines. Of course, the security of your key against theft will then be bounded by the security level of the weakest of these machines. You have to balance the convenience of being able to use many machines to read your emails, with the increased risks.
To move the private key safely (independently of the wisdom, or lack thereof, of having the same private key in many machines), you should protect your "keyring" with symmetric encryption and a passphrase of high entropy. Encryption of the keyring with a passphrase is the normal setting, so your job is "just" to use a strong passphrase. Since the format allows for reasonably strong password hashing (with iterations and salts), you can get away with a passphrase of entropy 60 bits or so. With such a strong passphrase, the keyring is "inherently" strong, and the travelling key is no longer an issue.
Each additional computer that has access to your keys increases your risk of being compromised. However, one way that you can greatly reduce the risk is by using a cryptographic smart card. Cryptographic smart cards store the private keys and do the encryption / decryption on the card. This means that the computer never sees the private keys.
Assuming you are using a smart card, if someone compromises the computer, they still can't access your private key. As long as it is plugged in, they can make use of it, but they can't steel your private key. Once you unplug the smart card, they have no more ability to decrypt or sign content.
Note: this only applies to cryptographic smart cards. Some simple smart cards don't do on-board processing; they offer little security.