It boils down to whether there is a chance your authorization code might leak.
PKCE is about preventing leaked "authorization code"s from being useful. When you ask for an authorization code, PKCE requires you to send the hash of a nonce. When you redeem the code, you must provide the original (unhashed) nonce.
This means that a malicious actor (XSS or CSRF vulnerability) observing the authorization code request and response cannot redeem the code, as they don't know the nonce.
Edit: Additionally, the client_secret means that only your server can redeem authorization codes. But, anyone can send messages to your server! If the authorization code has been leaked, then that means you don't know who is sending your server the authorization code -- is it the person who initially asked for it or some eavesdropping app? Without checking the nonce, your server may end up using its client_secret to redeem authorization codes for an eavesdropper.