I'm concerned about opening attached Word documents and other files (images, PDFs, etc). Gmail has a feature that previews these in the browser for you in a popup, without leaving the Gmail screen - it looks like they've rendered images of the document in some way and are showing these instead. Is there any risk of malware if you view documents this way?
It will prevent any exploitation of any vulnerabilities unique to the application that you would use to normally open the attachment (e.g. Adobe reader, image viewer, etc).
It will not prevent exploitation of vulnerabilities in the browser due to rendering the attachment (although this would hard as the attacker would have to create a browser exploit that works with the specific implementation of the previewer that Google are using).
And it will not prevent exploitation of vulnerabilities in any algorithms or libraries shared with the previewer and your usual application. However, for the preview, again the attacker would need to craft the attachment in a specific way to not only exploit the library, but to have this exploit then generate the appropriate browser exploit. The browser exploit could take the form of a simple attack vector such as an XSS attack though, which would be easier to achieve.
When you open an attached document on Gmail using the preview mode (if I may say so), you are safer because it is just an image of the attachment; you can imagine this image as a screenshot, I mean opening the document that way makes you safer because it does not include the functionalities of the original document:
However, a similar functionality is offered by Google Mail Checker Plus extension, it previews a file in a pop-up window as you described it. Previewing the file that way does not expose you to a risk, just like in the functionality described above, however (a case that reflects @shroeder's comment to your question) I reproduce to you an example of vulnerability that was exploited in the past allowing an XSS attack:
Lets look on popular (18,368 installations per week) extension called "Google Mail Checker Plus" [#]. This extension simply displays the number of unread messages in your Gmail inbox, can make preview of mail and supports desktop notifications.
Mail preview in popup of Google Mail Checker Plus
On preview we can see at least subject, from and piece of body of letter. Ok, lets send to us self letter with subject like: ::
document.body.innerHTML = ''; img = new Image(); img.src = 'http://evil.com/stallowned.jpg'; document.body.appendChild(img);
For the first we see such notification:
XSS in Google Mail Checker Plus notification part
For the second we click on extension's icon and see our worked payload in popup window:
XSS in Google Mail Checker Plus
It works! By the way this XSS has been already reported_ by Lostmon in June 03, 2010 and fixed version of extension is available. Lostmon wrote that: "All extensions runs over his origin and no have way to altered data from extension or get sensitive data like , email account or password etc.."
Let's discover this web vulnerability in context of this extension to understand risks.
.. [#] https://chrome.google.com/extensions/detail/gffjhibehnempbkeheiccaincokdjbfe .. _reported: http://lostmon.blogspot.com/2010/06/gmail-checker-plus-chrome-extension-xss.html .. _notifications: http://www.html5rocks.com/tutorials/notifications/quick/