If you do a DDoS by sending large amounts of traffic to that site, you're very likely creating a lot of collateral damage since other services in (parts of) the network will suffer as well if the network is saturated.
Also, very often phishers use hacked websites (for example poorly managed and outdaged Wordpress installs) to host their phishing sites, so you're not just attacking the phisher, but also a (mostly) innocent victim of that phisher.
And as others pointed out, just as in 'the real world' (which this is just as much a part of), you shouldn't take matters into your own hands.
The right thing to do, is either complain to the owner of the site or the network hosting it, or report it to the website being phished. Especially banks often have dedicated teams (or hire companies) which are specialised in taking down phishing sites.
In addition: you must consider that, when you are DDoS'ing a website, you're not attacking the web "per se" but the whole server, so you're causing the damage to the webhosting server (that may propagate among other websites hosted in the same server).
Finally: Most laws in most countries consider illegal to send any cyber-attack, it does not matter if it is against a legal or illegal target.