An antivirus program is always useful. The problem with your logic is that you're assuming that you are in control of everything that is downloaded to your machine. Using web applications will invite all sorts of different possibilities into the mix.
Lots of different things are downloaded to your machine without your knowledge. I just recently visited a site for holiday shopping and
SilverBells.wav was immediately downloaded to my machine without any kind of prompt. Data masquerading as session cookies, images, all are downloaded without prompting to the user.
"But I don't run the browser as an administrator, how can malware do any damage without privileges?"
It turns out that Malware really doesn't need very high privileges to get its job done. Any kind of initial foothold into the system is all it needs to run rampant. It might not be super advanced, and hard to remove. But the annoying Adware types usually don't care all that much.
Browsers are large applications with tons of attack vectors for exploitation. See my answer here that includes links to the main browser security bug fixes. You don't even need to visit a malicious site if an ad somewhere contains a piece of code to exploit the browser. In these cases files can be downloaded to your computer, or browser extensions can be installed; all without your knowledge or consent.
Nowadays antivirus products are a lot more than just scanning your system for malicious pieces of code. A lot of times now they provide real-time web browsing analysis. If a malicious script is running on a web page it will alert your first before displaying the web page itself.
They can also scan for malicious URLs and possible phishing attempts. Just because it's displayed on Google doesn't mean that you should click it. Some phishers will do a lot to push their website up the chain of Google returned searches.
The bottom line is that there is a lot more out there to worry about than just downloading files when it comes to malicious pieces of code. Using an antivirus program is always suggested; better to have it and not use it than unprepared.