From @Schroeder's comment, I made some additional searches and found out much more about this type of attack.
I Was the Target!
First of all, it looks like I am the target of the attack. Why? Because the DNS Amplification attack means that the source port of the DNS requests is set to 53. This makes it possible to force my DNS to send a non-requested response to another server (see graph below). Since I have less than 0.1% of such hits using that port, I don't really think that my DNS was used for the Amplification, but rather that it was the target.
How does the attack work?
The attacker sets up a computer with software sending UDP packets to some random DNS servers asking for a domain name. The packet looks normal except for the fact that the attacker puts the IP address of a different computer instead of his IP address as the source. The result is that the DNS sends the reply to the wrong computer:
10.0.0.1 10.0.0.2 10.0.0.3
+------------+ +------------+ +------------+
| | | | | |
| Attacker +-->| Some DNS +-->| Me |
| | | | | |
+------------+ +--+---------+ +------------+
| ^
v |
+---------+--+ This step is not required, it happens if
| | your DNS is setup to accept recursive
| Master DNS | requests (which is not a good idea)
| |
+------------+
10.0.0.4
In the example above, the attacker's request should have 10.0.0.1 as the UDP origination address. But instead, the attacker changes his IP address with 10.0.0.3. As a result, when the DNS at 10.0.0.2 receives the UDP packet and is ready to send a reply, it sends the data to 10.0.0.3.
The Master DNS of the domain is going to be used if the DNS at 10.0.0.2 doesn't know anything about the domain name being queried.
IMPORTANT NOTE: This is true of all UDP services. DNS is probably the worst in terms of amplification, but NTP, for example, can also be targeted.
Why not just verify the Source Address?
It's not possible from 10.0.0.2 because UDP has no recollection of the real source of the packet.
What is possible is for ISPs to verify that all the packets going out of a given computer has the IP address of said computer. Some do, I think, but most probably do not. It has a small speed impact... which of course with a DNS amplification is to laugh at (the DNS amplification has a way worse effect on the Internet than a small check of UDP packet origins).
The other thing may be that a user may still be able to do by connecting to the Internet in such a way that he bypasses the ISP verification. I do not know whether and/or how that would be possible, but I wouldn't be surprised that it could be done.
Actually, this is very problematic since the origin of such attacks is difficult to track and that's certainly why they are still occurring en masse.
Why is that a DDoS?
The packet to request a DNS record, which is what happens here, is very small, maybe 300 bytes. So the attacker only has to send one small UDP packet.
The response, however, may be multiple kilobytes of data. Specifically, there is a domain which is being used in these attacks:
peacecorps.gov
and that domain returns over 3Kb of data! (It defines many 'TXT' fields which are used in that amplification attack.) This is why it is called Amplification Attack.
As a result, the requests from the Attacker are transformed in about 10× bigger responses and the computer being attacked (10.0.0.3) gets swamped with a large number of rather large UDP packets.
Here I show the resulting entry in iptables. The first number represents the number of hits to my DNS computer after about 40 minutes (so over 10,000 hits a minute...):
pkts bytes target prot opt in out source destination
61637 4376227 DROP udp -- eno1 * 0.0.0.0/0 0.0.0.0/0 udp dpt:53 STRING match "|0a7065616365636f72707303676f76|" ALGO name bm TO 65535
Notice also how the string was fully transformed to hexadecimal numbers.
Wouldn't an HTTP request be even worse?!
HTTP 0.9/1.0/1.1/2 use TCP which is a two way protocol and no amplification can be generated with TCP. i.e. TCP breaks if you did not first properly connect with the full handshake.
HTTP/3, though, introduce the QUIC protocol which is HTTP over UDP packets. So far I have not heard of major issues with QUIC, but the protocol has changed a lot within the last 6-7 years and it is not yet widely implemented. Here is an article about the fact that QUIC has built in features to prevent amplification attacks, so there is hope that it has been taken care of.
Some people are talking about using HTTP to query for domain names. (DoH — Domain over HTTP). Hopefully that would not be implemented using QUIC...
How can I Fix the Issue?
The amplification happens because your DNS is setup to accept requests of domain names that you do not control (i.e. domains you're not the owner of).
BIND has an option that allows it to do amplification, which is called recursion in DNS parlance. This feature is now off by default, but you may have turned it on (by mistake?).
Here is the wrong setting:
allow-recursion { any; };
What you want is to use the correct settings:
trusted-servers { 192.0.2.4; } // list all your trusted servers
allow-recursion { trusted-servers; };
This will make your server automatically deny all such requests. So if you are not peacecorps.gov and you receive such a request, BIND will just stop right there and write a note in your DNS logs about the denied request.
Could I block the IP from those requests?
Yes. I started by doing that because my server was running well over 100% in CPU time. However, it may not be wise to do so. From the picture above, you can see that your DNS server sits between an attacker and a victim. If you block the source IP address, it's not the attacker's IP that you are blocking, it's the victim's. This means you're actually preventing the victim from seeing your domain names and do legitimate requests. That's probably not what you want!
At first, I generated a log message from my firewall. If I were to detect 5 or more requests to port 53 (UDP) from the same IP address in a short amount of time (5 sec.), then I would block the IP address. To do so, I used fail2ban.
First, I have a filter which detects the links with UDP and port 53 as the destination:
# Filter: /etc/fail2ban/filter.d/named-fast-requests.conf
[Definition]
failregex = \sIN=[a-z0-9]+ .* SRC=<HOST> .* PROTO=UDP .* DPT=53\s
Second, I have a jail which gives the other parameters:
# Jail: /etc/fail2ban/jail.d/named-fast-requests.conf
[named-fast-requests]
enabled = true
filter = named-fast-requests
action = named-action[scheme=all,period=year,reason=named fast requests]
logpath = /var/log/iptables/iptables.log
maxretry = 5
findtime = 5
bantime = 1036800
The main point here is the maxretry and findtime which are set at 5 times in 5 seconds or less. When that happens, I block the <HOST>.
You will want to update the action with your own thing. I use my own iplock tool here. The command I use in the action:
actionban = /usr/sbin/iplock -s named -b <ip>
By default, fail2ban uses iptables directly. Search their actions to see how it's done.
Don't Be the Source of Amplification
You should block requests where the source UDP port is less than 1024. These are not valid. If you offer a server, port 53 will be a destination port, not source.
iptables -I INPUT 123 -i eth0 -p udp -m udp --sport 0:1023 -j DROP
WARNING: replace the position (123) with the correct number!
This rule says that for any incoming UDP packet on eth0, which has a source port between 0 and 1023, drop the packet. This prevents someone from using your DNS server for amplification. However, it does not protect your own server from amplification from others. Note that even with the proper allow-recursion setup, you would not prevent an amplification attack if the attacker properly selects one of your domain names for the attack.
Note: if you specify a different nameserver to resolve domain names on your computer, you may want to open those connections before you block everything from port 0 to 1023. This would be something like so:
iptables -I INPUT 123 -i eth0 -p udp -m udp --sport 53 -s 8.8.8.8 \
-d <your-static-ip> -j ACCEPT
This is the returned data from the nameserver 8.8.8.8, which you probably want to receive. It is not likely that your provider or some other official nameserver would be the direct source of a UDP attack. If you do not have a static IP address, you do not have to include the -d option.
However, it is probably a lot better to use the ESTABLISHED feature which is now available for UDP messages (it has been for a while now, but I remember a time when that was not available...):
iptables -I INPUT 123 -i eth0 -p udp -m state \
--state ESTABLISHED,RELATED -m udp -d <your-static-ip> -j ACCEPT
This means you'll automatically accept answers from your domain nameserver provider since the firewall will know you sent a request and you want to accept replies. This rule has to appear before the DROP rule above.
Drop Unwanted Requests Early
Obviously, having BIND accept all those requests with peacecorps.gov is not something anyone wants. You can actually block those directly in your firewall. This works because UDP packets are not encrypted so the domain name is visible.
Here is a rule one can use to block those domain name requests:
sudo iptables -I INPUT 123 -i eno1 -p udp -m udp --dport 53 \
-m string --hex-string "|0A|peacecorps|03|gov|" --algo bm -j DROP
Obviously, if your DNS has any domain or sub-domain including "peacecorps.gov" then it should not use that iptables rule. For most of us, though it should be rare.
The --hex-string option allows you to specify a string. The way it is defined in the UDP packet uses a form of P-string (size + data). Since "peacecorps" is 10 character long, we put 0x0A just before it. Again, the "gov" is three letters so we use 0x03. If we were to use the string "peacecorps.gov", it wouldn't work since the period would not match the 0x03 byte. The first size is optional, though (although you'd match anything that looks the same such as "bestpeacecorps").
Having such a rule will save your domain name service a ton of totally unwanted traffic.
Update: Although the attack stopped about two weeks after I posted my question, the "peacecorps.gov" issue still happens about 10 times a day.
Source: https://defragged.org/2020/05/20/tips-and-tricks-blocking-dns-requests-via-iptables/
Debugging You DNS
In order to see what query your DNS server receives and replies to, you can run the following commands in your console:
sudo rndc querylog
Now look at the logs, usually here:
less /var/log/named.log
Look at the bottom (hit G in less) and you should start seeing queries from remote IPs. The logs include the domain name being checked. It's very practical, especially if you missed entering some of your own domain names in your secondary or tertiary DNS.
