Isn't this just allowing anyone to access your API regardless of
whether they have the secret key or not?
Yes, The implicit grant lacks the ability to authenticate the client, which the other grants can do — further introducing attack vectors that the authorization grants, which require a client secret, do not experience.
If client authentication is important then a different grant must be implemented.
Implicit grant is more secure in the sense that it wouldn't expose the client secret, which can be shared across your internal applications.
Here's a good read on when to use which (OAuth2) grants and OIDC flows https://community.apigee.com/articles/41719/when-to-use-which-oauth2-grants-and-oidc-flows.html