The attacker can use a vulnerability on the browser to execute the malware code on the client's computer. Most malware that I seen obscure their shell code using Base64 and compression inside a JS file or embedded into the HTML. Once the JS code is executed on the clients computer, it uses 'deflate' and 'eval' to execute whatever code is encoded on the string.
The malware takes advantage of a bug in the browser to execute shell code on the clients machine. Buffer overflows used to be quite common, but these days major browsers tend to be written with more care on that point. Use-after-free and double free are still pretty lively.
If you are getting multiple random pop-outs from legitimate website, this may be an indication that the computer is possibly infected with something. Make sure that your AV and Anti-Spyware solutions are up to date and running correctly. Also, check whats running on the computer using tools like process explorer and autoruns.
The JS can continue this even if the user cancels, effectively making the user's browser unusable (they may not know how to kill a process manually) and often attempts to make the download look like antivirus software.
Some proportion of users will accept the download just to get their browser back, and then the malware vendor just has to wait for the binary to be run.
The core problem with viruses and malware is that the OS confuses the current user with the programs they run. When I run Solitaire, that process has the ability to edit every file that I can from the command line or Windowing system. That should not be the case.
There's no reason why Solitaire or a screensaver that you download from the web needs access to your tax files, so you can use a combination of cues to figure out what privileges it needs
Other privileges can be similarly granted based on user designation or simple rules.
Most operating systems and applications were not written with this in mind, so systems like Polaris from HP labs shows how to allow apps to run with less than current-user permissions on Windows.
It's been difficult for browser and system makers to make their legitimate messages hard to counterfeit. Windows User Account Control is one of the best techniques because of how it interrupts every other program when it needs privileges to complete a task. Most browser and software pop-up messages are easily faked and you should be wary of them.