I've been told that when integrating with a 3rd party payment gateway, the other website's window should not be opened in an iFrame or a separate window - that the current application window itself should redirect to the payment gateway.
These are not PCI requirements.
You should read the PCI DSS E-commerce Guidelines. Section 3.4 describes "Common E-commerce Implementations" such as:
and, to quote,
These examples are intended to be representative of only a few of the most commonly found basic implementations. By no means are they meant to cover the vast range of deployments, hardware components, software applications, and hosting/services models that may exist.
Number 3 is what your contact has told you you are required to do. They are incorrect if they claim that PCI requires you to be limited to that, obviously. They probably mean that's what they support and you should do it and waving around PCI is an effective bully stick.
As a rule of thumb, any business partner that tells you you must do something because of PCI should be able to quote line and verse at you. List which item of the PCI DSS or SAQ that they feel you're in danger of violating. That gives you the information you need to resolve the issue. A business partner who says "PCI says" but can't or won't say where is usually lying or confused.
(Sometimes you have to dig for it; often policies get determined by people who don't deal with customers, but get propagated by support and account management personnel, who don't know what drives the requirements. But you should always be able to ask for an answer from somewhere up the chain).
Opening a new window or iframe does not change the protections provided by the Same-Origin Policy. The origin inherit ice rules for iframes prevents two different domains from accessing each-others data.
There is no security or PCI-DSS requirement that would require a redirect from/to a payment gateway. This maybe a usability requirement mandated by a specific payment gateway.
