"The Web Application Hacker's Handbook: Finding and Exploiting Security Flaws" by Dafydd Stuttard, and Marcus Pinto.
"XSS Attacks: Cross Site Scripting Exploits and Defense" by Seth Fogie
There are many possible attacks, which all depend on the JS engine, the available API and the execution environment. JS is a language after all even though I suspect you only are interested in the Web uses. I know next-to-nothing about Web security, but for instance JS is also part of the PDF format
5 years ago you could load DTD's in a XML document in your JS code on Acrobat Reader, and make an external entity of your DTD point to local files on the system so you could get to read them if they were XML-like or plaintext. Obviously this was an unintended consequence of the API, which was changed. No engine bug in that cause though. Nowadays the same attack with the same API would also be impossible because the execution environment changed: Acrobat Reader now sandboxes the PDFs it opens (in Protected Mode).
In short, even though I can't really answer your question I can still tell that someone who doesn't cover all of this is not properly answering:
Most of the Vulnerabilities in the browsers are in the complex parts, such as the dynamic DOM-tree building.
The other thing is like the beef exploit framework, uses the tricks the browser allows you to do (like Steve DL notices).