If the operator of the Pwned Passwords API is malicious (or the service is hacked by a malicious person, or someone intercepts your communication) it can lie about certain passwords (only return a subset of passwords it knows for that hash), record where the request came from, identify the website belonging to that ID, identify the account (based on creation time, for example) and test it with the passwords it lied about, for a decent chance of account compromise.
This is a fairly unlikely scenario (the owner of the API is a respectable security professional, the list of known hashes is public, and doing this kind of attack without being detected would be hard), so it depends on how risk-averse you are. If you run a discussion board, using the API is definitely a good idea. If you run an e-bank, maybe not so much.
Downloading the database and doing the checks locally is of course safer, but not a trivial task, given the size.