The OpenIDConnect states that id_token is for the client (relay party) which would like to be sure that user is authenticated. But what does it mean? If you will take a look into how the id_token is generated (the authorization code flow with PKCE - this is the most recommended option currently), you will see that client (replay party) is generating some values (hashes) which are send to identity provider and after successful authentication are included into id_token (after some math operations). Those values are used to validate id_token by client, to check if it was generated (and send) by correct identity provider. Those values generated by client are also stored temporarily on client and then used to validate id_token.
Now the problem is that when you are sending id_token to resource server (API), it cannot properly validate it, as API do not have all required values to perform this process. For me that is the main security reason. (of course it is still dependent on the implementation).