Yes, WebGL is indeed a potential security risk, though the magnitude of the risk is hard to assess and open to debate. There are some tricky issues here. The browsers have put in place some defenses against the security risks, but there seems to be some debate about whether those defenses will prove adequate in the long run.
One major risk is that WebGL involves running code directly on the video card, and exposing APIs that provide direct access to video card APIs. The browser does attempt to sandbox this code (to a certain extent), and browsers do enforce a number of security restrictions designed to prevent malicious behavior. However, many of these APIs and their implementations were not originally designed to be provided to untrusted entities (they were only usable by native applications, which are fully trusted), so there are concerns about whether exposing them to arbitrary web sites might enable web sites to attack your system.
There was one high-visibility white paper (see also the sequel) which looked at the security of the WebGL implementation in browsers at the time, and found a number of vulnerabilities. They found some memory safety issues in several WebGL APIs, and also found some attacks that would allow one web site to read pixel data of other web sites (which could enable a breach of confidentiality). See also this third study, which demonstrated the existence of these vulnerabilities on a number of browsers and web cards (at the time).
Browsers have responded to this with a variety of defenses: they have blacklisted video cards with known security problems; they have tried to fix the known memory safety problems; and they have restricted use of WebGL per the same-origin policy, to prevent a malicious web site from using WebGL to spy on users' use of other web sites.
There is some ongoing debate over whether these defenses will prove adequate in the long term. Microsoft has taken the position that WebGL is too great a security risk and the existing defenses are not robust enough. On the other hand, Mozilla takes the position that the defenses they have put in place will be adequate, and that WebGL provides important value to the web. Ars Technica has an excellent round-up of the issue; and here is another press report.
P.S. I completely disagree with your statement about it being particularly a problem for open source web browsers. That's a myth. See Open Source vs Closed Source Systems, which already covers these arguments. (See also Chrome vs Explorer - how to explain in plain words that open-source is better? for additional thoughtful discussion on this topic.)