edit: since you stated that you already have the smartcard working there is no need to expand on the ssh-agent.
solutions for a smartcard can be using agent-forwarding or (with and without sc) using a proxycommand for ssh.
You could use a dedicated configuration for this in ~/.ssh/config by using a configuration for host B, or use it as a parameter in an ssh invocation.
ProxyCommand ssh -p 22 -q -W %h:%p <the-real-hostname-for-A>
an advantage is that you will have an alias for the host and also have code completion(less typing)
TAKE CARE: agent forwarding is a bad practice on a shared machine, since it is extremely easy to hijack your agent via the environment variable SSH_AUTH_SOCK so that an attacker can take over your identity AND authentication mechanism to hop to other hosts.
When using yubikeys as your smartcard, you can use the feature that you need to physically touch your smartcard when a cryptographic operation is required like logging in. this way, the attacker also needs physical acccess to your smartcard, twharting the attack in most cases.