If I have DNS over HTTPS and DNS over TLS activated simultaneously (router has DoT activated and smartphone browser has DoH activated, so I see on https://18.104.22.168/help DoH: yes and DoT: yes), which one is used?
Both are services offered by some parties, like your router.
Which one is used? Both, either, or none. This all depends on the applications and the OS. They could even do DoH/DoT towards other resources externally.
AFAIK browsers do DoH, in different ways (automatically or not, only for some nameservers or not, etc.), but not DoT. The underlying OS and/or local resolver may be using DoT as forwarding for example to other external resolvers.
then I wonder how the site 22.214.171.124/help can even detect that my rooter has DoT enabled
Most DNS tests on the web do use the same technique: include (through an image for example) a link to a resource on an hostname you control, and it being random and unique. Then as you control the authoritative nameservers you can watch if requests do come (and since you are using a random name, you are not influenced by DNS TTLs and caching) and control them somehow (like a request expected to be DoH would only be served over DoH endpoint and not over standard DNS, etc.)
For this page if you look at network traffic you see requests being made to names like:
I think the names themselves do hint on how they are used in the test.