openssl dhparam has a
-check argument you can use. But the current version of openssl is incorrect and is too strict. As evidence, the DHE groups from RFC 7919 which are the only groups you can use for DHE with TLS 1.3 do not pass this check, even though they are safe to use.
The "unsafe" group params are those generated by
openssl dhparam -dsaparam. They are supposedly actually safe to use as long as your TLS server has
SSL_OP_SINGLE_DH_USE set (as the security advisory you linked says). With OpenSSL 1.1.1, this option is always on and does nothing (compare the manual for 1.1.0 and 1.1.1).
You can read the code invoked when you run
dhparam -dsaparam and
dhparam -check. Just don't ask me to explain why the code does what it does.
On Security Stack Exchange, people can read man pages and browse code on Github, but not necessarily really understand the math. If you want someone to explain the math to you, you should ask on Cryptography Stack Exchange. But here is an answer that might be enough to satisfy you.
What you should do: either replace the DH group your webserver uses with one generated by
openssl dhparam 2048 or use the values from RFC 7919.
To answer the question "how did the pentester know?", the pentester likely ran something like this:
rm -f dh.pcapng
sudo tcpdump -w dh.pcapng &
echo "Q" | openssl s_client -cipher 'EDH' -servername $1 -connect $1:443
sudo killall tcpdump
printf "asn1=SEQUENCE:seq_sect\n[seq_sect]\nfield1=INTEGER:0x" >dh.cnf
tshark -r dh.pcapng -2 -R ssl.handshake.p -T fields -e ssl.handshake.p | sed -e 's/://g' >>dh.cnf
printf "\nfield2=INTEGER:0x" >>dh.cnf
tshark -r dh.pcapng -2 -R ssl.handshake.g -T fields -e ssl.handshake.g | sed -e 's/://g' >>dh.cnf
openssl asn1parse -genconf dh.cnf -noout -out dh.der
openssl dhparam -check -noout -inform DER -in dh.der
with a domain name as argument (put that into a file called
dh.sh, then run
bash dh.sh www.example.com with your domain).