It depends on the cryptosystem and its implementation: there's no guarantee that the computer can tell a correct decryption from an incorrect one.
However, most systems designed for static data encryption (that is, for example, file or e-mail encryption, as opposed to just temporarily encrypting data to transfer it over an untrusted network) do include some way to detect when an incorrect key has been given. The exact mechanism varies, but some commonly used ones include:
Check the key before attempting decryption, by including a cryptographic hash of it as part of the encrypted data. (For messages encrypted using public-key cryptography, one could simply include the public key, or a hash of it, in the encrypted message.)
Check the key at the beginning of the decryption, by prepending a known header to the plaintext data — this could be as simple as a block of null bytes — and checking that it decrypts correctly.
Check the key — and the integrity of the data — by including a message authentication code in the encrypted data (or using an authenticated encryption mode). The main purpose of a MAC is to protect the encrypted data against tampering, but as a side effect, it also serves to detect incorrect decryptions. However, the MAC cannot be verified without first reading and processing all the data, so for the sake of usability, it may still be useful to combine this with one of the other key verification methods described above.
In any case, a well designed encryption format will allow the user to be quickly notified, if they try to decrypt it with the wrong key — but not too quickly, as that would just make it easier to crack the key by brute force without doing anything to help legitimate users. The recommended way to achieve this, for symmetric encryption with a key derived from a passphrase, is to use a key derivation function (such as PBKDF2 or scrypt) with an adjustable parameter that controls how much work is needed to compute the key from the passphrase. This derived key (which should be long enough — say, 128 bits or more — to not inteslf be crackable by brute force) may then be verified with one of the methods described above.