When comparing with a different card reader it is immediately obvious a permission problem.
$ getfacl /dev/bus/usb/003/021
getfacl: Removing leading '/' from absolute path names
# file: dev/bus/usb/003/021
# owner: root
# group: root
user::rw-
user:sanmai:rw-
group::rw-
mask::rw-
other::r--
Where for the problematic card reader there's no extra user permission:
$ getfacl /dev/bus/usb/003/020
getfacl: Removing leading '/' from absolute path names
# file: dev/bus/usb/003/020
# owner: root
# group: root
user::rw-
group::rw-
other::r--
Under Debian-based these extra permissions are set by udev rules:
cat /lib/udev/rules.d/60-scdaemon.rules
These rules can be extended by adding a line to, say, /etc/udev/rules.d/60-scdaemon-local.rules
with vendor and product IDs of the problematic reader:
SUBSYSTEM=="usb", ATTR{idVendor}=="xxxx", ATTR{idProduct}=="yyyy", ENV{ID_SMARTCARD_READER}="1", ENV{ID_SMARTCARD_READER_DRIVER}="gnupg"
Then reloading the rules:
udevadm control --reload-rules