Is "sign in with Facebook", "sign in with Google", etc. bad for security? A hacker only needs to compromise your Facebook/Google/Yahoo/etc. account, and they'll have access to all of your other accounts that are connected to your Google or Facebook account.
Most of the websites that potentially use "Sign in with Google" type features, use your email address as a fallback to reset forgotten passwords. For all these websites, your email address (where ever that is; be it gmail or some other provider) is a single point of failure.
In fact, if you use a Google account with say 2-factor authentication and strong passwords it's quite good for security, as now effectively everything uses 2-factor auth with a strong password.
It also makes is a better user experience as you only need to remember one strong password, versus remember unique strong passwords for every web service you sign up for. (You should not reuse passwords as you are not sure if some web services are malicious/incompetent and record your passwords/password attempts in cleartext or with weak hashes).
(The major exception to the email based password reset in my experience is banks, where you typically have to talk to someone in person/over the phone and answer several security questions/provide documentation if you forgot your password. I do think it would be a mistake for banks to let you login through Facebook/Google, though I have never seen it in practice.)
I would say it was good for security because the password database is with a trusted company and you can pretty much rest assured that it will be stored in a secure format, rather than in a format decided at the whim of the site developers. Any SQL injection or other data leakage flaws on the website could allow access to your password and account.
only needs to compromise your Facebook/Google/Yahoo/etc. account
I'm not sure I agree with your usage of "only" here, implying that any access to such accounts would be trivial.
Email and social media accounts should be your number one priority to secure. These are "hub" accounts and any attacker that gains access to them can often use them to gain access to everything you own online. For example, an attacker could perform a password reset on your Dropbox account because they can now get access to the password reset email that is sent to you.
Bottom line would be to secure all your hub accounts with a strong memorable password, and to protect these accounts with two factor authentication.
Centralised login using OAuth or OpenID Connect is beneficial for security overall, as long as you trust the identity provider. Any compromise of your account can be shut down in one place, securing all sites using that authentication or authorisation method. It also discourages password reuse across different sites where any compromise exposes all accounts which then have to be secured individually.
However, sometimes implementation of these single sign-on systems can be complicated. And there is also the problem of authentication being confused with authorisation. Also, see this answer. Therefore, sometimes such implementations can introduce different types of session management vulnerabilities into an application.