If your server is a recursive server, block ALL queries from unknown/untrusted sources, regardless of source port. You should also block queries, even from trusted sources, with a source port of 53.
If your server is an authoritative server, yes, you should still block queries with a source port of 53.
In fact, you should block queries with any privileged source port, and even some unprivileged source ports. For instance, 5060 (SIP) is also a good source port to block.
Your server will send its answer back to the source IP and port. Due to the nature of UDP, both can be forged.
In a DDoS attack on a third party's nameserver, the attacker may forge the victim's IP, and the victim service's (DNS) port (UDP 53) and send a query to your server. Your server will then send the (much larger) answer to victim server, and unwittingly become a weapon in the DDoS attack itself.
The very vast majority of nameservers send queries from a random unprivileged source port.. The mere fact that a query purports to be from UDP 53 is itself a strong indicator that it is malicious.
Blocking in this manner won't do much to protect your server from being the victim of a DDoS, but will do help protect your server from being used to help perpetrate a DDoS