First of all, I'm not a QSA.
That said, because you're accepting credit card information directly on your site (even though you don't send it to your server), you need to fill out SAQ A-EP. Among the things you certify with that SAQ are:
- All processing of cardholder data is outsourced to a PCI DSS validated third-party payment processor
- Your e-commerce website does not receive cardholder data but controls how consumers, or their cardholder data, are redirected to a PCI DSS validated third-party payment processor
- Your company does not electronically store, process, or transmit any cardholder data on your systems or premises, but relies entirely on a third party(s) to handle all these functions
Do note that "cardholder data" does not include the billing address, even though it's data about the cardholder. In this context, it means:
At a minimum, cardholder data consists of the full PAN. Cardholder data may also appear in the form of the full PAN plus any of the following: cardholder name, expiration date and/or service code.
So there's no issue with posting the address back to your own site, so long as the card information doesn't go with it.