It is not possible to recover the key ... assuming the key has sufficient entropy.
Here's what the attacker can do. If the attacker has a guess at the key, the attacker can check whether his guess is correct. This means that if the attacker can narrow down the list of possibilities for the key to a short enough list, the attacker can simply try all possibilities in that list and see which one is correct, and the attacker will eventually discover the correct value of the key. This attack strategy is known as brute-force search, or exhaustive keysearch. It turns out that this is essentially the only attack an attacker can do; with SHA1-HMAC, an attacker can't do any better than this attack strategy.
So, your job as defender is to make sure that the number of possibilities for the key is large enough that brute-force search will be unsuccessful. Ideally, there would be at least 2128 possibilities for the key, all equally likely. For instance, you might generate the key by reading 128 bits from
/dev/urandom. If that is how you select the key, then there will be no way for an attacker to recover the key.
On the other hand, if you do a lousy job as defender of picking the key, then it might be possible for an attacker to recover the key. For instance, suppose you decide your key will be a random 4-digit number. Well, then there are only 10,000 possibilities for the key, so an attacker can try them all and identify which one is correct. The trial-and-error process will take only milliseconds -- so if that's how you picked your key, things would be totally insecure, and the attacker would be able to recover the key. If you pick a random 10-digit number as your key, then there are 1010 possible values of the key, which is still not enough: an attacker could try all of those possible value in mere hours. So, don't do that. Or, if you use someone's telephone number or SSN as the key, then again it will be possible for an attacker to recover the key (simply by trying all possibilities). So, don't do that, either.
In short: choose your key using 128 bits of crypto-strength randomness. Then, it will not be possible for an attacker to recover the key.