Signature and encryption are entirely separate things although they use similar technologies to accomplish their goals.
Signature is used to verify the identity of sender. If attacker has access to A's private key, he could create fake messages that appear to originate from A.
Encryption is used to scramble a message so that only receiver can read it. Typically the sender A would use the public key of the receiver B to encrypt a message, not the other way around as your question suggests. The message is encrypted using the public key of B, and decrypted using the private key which only B has access to.
This way both encryption and signing can be safely accomplished even if attacker has both A's and B's public keys. They are called public keys because they can be safely given out, shared and published, and only private keys need to be protected.
Your question implies public and private keys are used incorrectly (wrong "direction), which might be possible in some weird system, but it wouldn't be secure, and the attacker would indeed be able to read the messages.
Done properly, the sender A would sign the message using his own private key, and encrypt the message using B's public key. Receiver B would then verify the identity of the sender using A's public key, and decrypt the message using his own private key.