Yes, exposing the client secret would increase the risk of a token being compromised, but there are other pieces that would need to perfectly fall into place for that to happen. If we look a an example POST for a token in the Authorization Code Grant Flow:
POST /oauth/token HTTP/1.1
Implementation may differ in slight ways but let's look at a couple other pieces of information (in addition to the client_secret) that the attacker also needs to get a token:
code - The authorization code that is granted is typically single use only. Obtaining a code would mean having access to an authorized user's account and completing the authorization request prerequisite to this grant type.
client_id - While not a 'secret' per se, this is the registered applications id which would need to obtained and is not usually something that is guessable. (though I suppose if the attacker has the client secret, they also have the id? Seems odd to have obtained one and not the other)
The authorization code is definitely added security compared to implicit grant, but also like you suggested, Auth-code is much more secure because the token is not meant to be returned to the user-agent (browser). It's more commonly implemented in a backend/frontend application with the code/token exchange happening in the backend.