A clever attacker can set up a site so that if you click on it even once, within seconds he can take over your computer. Even worse, you might not have to go to his site. If you click a link that leads to his site, he could feasibly "own" your computer. Beware!
Whenever you visit a Web page, lots of invisible activity can happen on your computer. For example, if the Web site welcomes you with your own name, that's because last time you visited, it put a special text file called a cookie on your computer. If a stock ticker scrolls past or sports scores update automatically, the site could be executing a script on your computer. This stuff is normal. But the fact that your computer is willing to receive and execute instructions from a Web site is what hackers exploit.
Not all Web-based attacks are visible to the intended victim. Sophisticated hackers use tricks to hide what they're doing. But such sophisticated attacks are rare. Most Internet users pay no attention to URLs, so the average attacker doesn't use sophisticated attacks, most of the time he does not even know how to exploit your browser, he would need a rare browser 0-day and this type of bug is worth hundreds of thousands of dollars, sometimes millions.
You probably haven't been infected just by clicking on the link.
I would not worry if you did not enter personal data on the phishing site.