3

My organisation has 200+ websites each with an assigned IP address, these IP addresses evaluate to a pair of load balancers. Behind these are a firewall and then 8 application servers behind this, all identically configured, same OS, identically patched, same directory structure and web sites/code and so on.

We have 5 base web applications on each of the app servers with certain IP addresses going directly to a specific application, the majority of websites go to one specific application which then applies a front end skin using a configuration file based on the web address being requested by a users browser.

Our IT dept. are now taking the decision to simply provide our ASV scanner with 5 IP addresses which evaluate to one of the 5 base applications, as we scan weekly the likelihood of scanning all 8 app servers is quite high in their opinion, the argument being even if we scanned all 200+ IP's there is no 100% certainty that we would scan all 8 servers, depending on the load we could hit the same one each time.

My argument is that we should take the PCI DSS requirement 11.2 and supporting guidance in the ASV programming guide literally and provide all 200+ IP addresses, however our IT department do not buy this despite being in black and white, we self assess and do not have a QSA, is there any definitive answer or argument I can use to ensure we submit all of the IP's?

2
+50

Okay, so I think we can actually tease this matter out into two issues. First, there's the compliance question of whether the PCI requirements compel you to scan every single one of those roughly 200 public-facing IPs, even if they just point to a far smaller number of actual web app instances on the eight servers running them. Second, there's the matter of how you can make sure you actually get those web/apps & servers scanned with 100% certainty.

Regarding the question of whether you need to scan all almost-200 public-facing IPs, based on some specific guidance in the "Localized Load Balancers" section pasted below it looks like you do actually have an "out" against doing so.

On page 17 of the above-mentioned guidance (mentioned in other answers & comments also) for how an ASV-certified scanning authority must conduct its scans, this passage occurs:

Account for Load Balancers

If a scan customer has deployed load balancers, the scan may only see part of the configuration beyond the load balancer. In these cases, the following applies: 

Localized Load Balancers:

The ASV must obtain documented assurance from the scan customer that the infrastructure behind the load balancer(s) is synchronized in terms of configuration. If the scan customer is unable to validate a synchronized environment behind their load balancers, the ASV must disclose the inconsistency with the following Special Note on the scan report: “Note to customer: As you were unable to validate that the configuration of the environment behind your load balancers is synchronized, it is your responsibility to ensure that the environment is scanned as part of the internal vulnerability scans required by the PCI DSS.” (Special Notes do not cause a scan failure or supersede any established CVSS scoring.) 

External Load Balancing Services:

The ASV must take into account the use of load balancing services external to the scan customer’s environment that direct traffic globally or regionally based upon source IP address location. Depending on implementation, external load balancing services may direct the ASV scan tools to only a regional subsection of a scan customer’s environment. Thus, the ASV scan solution must accommodate external load balancing scenarios to ensure that all IP addresses and ranges provided by the scan customer are successfully scanned.

The use of load balancers, the configuration, and the customer’s assurance must be clearly documented in the scan report.

(Blockquoting the above screwed up part of the formatting a little in the cut-and-paste vs. the original pdf, but the text of the content and the meaning of it remain completely unchanged.)

The only sensible interpretation I can give of that text is that scan customers who do their own load balancing merely have to certify, to the ASV's satisfaction, that all the servers you have behind the load balancer/s are "synchronized", so that you can say with a high degree of confidence that any ones that do happen to go unscanned because the load balancer/s just don't happen to forward traffic to them during a scan are in the same state as the ones that are successfully scanned. The language saying that entities in a localized load balancer situation don't have to scan every single externally-available IP is expressly different from the language that covers situations for non-localized load balancers; this implies the guidance authors knew and intended exactly that firms who use localized load balancing don't have to scan everything. Similarly, while the meaning of the rules for firms running localized load balancers here does contradict the repeated command in every other part of the ASV guidance that every external IP must be scanned, this section clearly sets down specifically-applicable guidance about what firms that have localized load balancers are allowed to do, which obviously overrides the more general guidance statements elsewhere. So, this must be exactly the "out" your acquirer was referring to, and to me it looks like they found a legitimate one.

What does keeping scanned and potentially-unscanned servers in "synchronization" mean? I agree with commenter AndyMac: it's likely the configuration images would have to be the same, or essentially the same. In terms of what a system looks like to maintain and prove that synchronization, I think a combination of using ordinary deployment tools carefully to ensure that the images you deploy are uniform + the use of change monitoring/comparison software to verify that everything remains as it should be would get the job done. But that's just a recommendation for one idea to look at; I'm sure there are other approaches.

One more important note: The text of the guidance above says if you can't show proper synchronization the ASV must put a little nastygram on your results saying so, but you don't fail the scan. Despite the possibility that some publically-reachable web apps & servers were missed by the ASV because of the load balancing, you are allowed to continue on in compliance with just an admonition that you really have to make sure to get everything scanned with your own vulnerability-scanning tools. So, yeah, unless your acquirer raises a stink it appears that not scanning all external IPs isn't a showstopper, even if you don't provide objective proof that any unscanned, publically-reachable web apps/servers are properly synchronized with scanned ones. (Ah, the security-first PCI council....)

So, moving on to quickly tackle the second point: actually getting all your publically-facing web apps/servers scanned. Really, you would like to get this done via the ASV. That saves you the hassle of either (a) implementing synchronization and documenting that for the ASV or (b) taking the little black mark on your scan results and relying solely on your own vulnerability-scanning tools and procedures to work perfectly every scan and catch everything that an ASV would catch. And the thing is... I really think you should be able to get the ASV scans to properly check each and every server (and all services & web apps available on each). I'll confess that I have little-to-no direct personal experience configuring load balancer-setups. But couldn't you could just: (1) set aside a public IP address for the ASV to use that points directly to your firewall, bypassing load balancing; (2) configure your firewall to forward all valid incoming traffic (ie. the traffic that you normally allow through the firewall) directed to that IP address to Server X, with Server X being the particular server (with the services/web apps it's running included) that you want to target on any particular scan, and; (3) one scan after another, methodically change the server that gets scanned from that one IP until you're run through every publically-reachable thing you have?

Or you could go web app instance-by-web app instance, perhaps. (Although in that case you'd have to do 40 separate scans rather than 8, I suppose.) Point being: there has to be a way to allow the ASV to just bypass the load balancing and directly scan your setup. Like you said, you already do have some fixed, direct, "IP X goes to web app instance Y." rules in place. So, actually, you are already doing this to some extent. You should be able to just build on that, except changing the particular web app a public IP connects to from one scan to another. (I think.)

Anybody who does have some direct experience working with external, third-party vulnerability scans in a load balancer scenario want to chime in?

Anyway, hope the above helps. Cheers.

(UPDATE: Rewrote the above answer on 10/06 as it reflected a misunderstanding on my part of some facts. Overall conclusions didn't change, however.)

6
  • The issue isn't that we may not scan all servers it's the fact we have 150 URL's each with an IP address pointing to one load balancer. We would have the same problem if we only had one application on one server, if we have 150 addresses pointing to 1 device loaded with 150 IP's why would we pay to scan 150 IP's when one would scan everything? Obviously we don't have one we have 5 duplicated on 8 servers, 3 of our websites go to 3 applications, the other 197 or so point to the same application with a config file determining what graphics and text is displayed based on the website address. Oct 6 '15 at 13:05
  • @MatthewParkes I think maybe we're having a miscommuncation situation here. So let me restate what my understanding of this is: You have eight servers, each running five web applications per server. It is these servers and web apps that, at the end of the day, are what you are trying to protect. To connect to the web apps on those servers clients can use c. 200 IP addresses, with load balancer/s then balancing the traffic across the web app instances and servers. However, you want to know if PCI reqs. mean that you must still have all of the external IPs scanned, correct? Oct 6 '15 at 20:24
  • Okay, on re-reading my answer to you I think I see some language on my part which might be responsible for your confusion. Will do a quick edit to fix that. But note that my analysis and conclusion doesn't change at all. That guidance section on load balancer setups is pretty definitive. Oct 7 '15 at 0:22
  • Thanks again halfinformed really appreciate your in depth responses, not sure whether our load balancers are localised, I presume they are, what we do is outsource the platform to a service provider based in London (datacentre). They manage the hardware, OS and services and the hardware is not shared with anyone else. We develop the applications in house and upload the code to a repository which is accessible by the service provider. Any changes we need to our platform be it hardware, software, services or code changes are made through a change request system. We link via an MPLS. Oct 8 '15 at 7:53
  • @MatthewParkes You're quite welcome. Actually, it was a learning experience for me as well. (If someone would have asked me off-the-cuff, without any ability to consult the guidance, I would have said "Oh, PCI definitely requires you to scan all public IPs. Definitely." I had completely forgotten about the load balancer exception.) Oct 9 '15 at 5:16
1

There's a document provided by the PCI SSC available @ https://www.pcisecuritystandards.org/documents/ASV_Program_Guide_v2.pdf which describes what should be scanned.

In brief, this states that all externally facing IP addresses must be scanned as well as FQDNs of all publicly available websites.

5
  • 3
    I'm sure the OP has read this: "My argument is that we should take the PCI DSS requirement 11.2 and supporting guidance in the ASV programming guide literally and provide all 200+ IP addresses, however our IT department do not buy this despite being in black and white" Sep 30 '15 at 7:31
  • Thank you AndyMac, yes I have read this document, we have also sought advice from our acquirer who agrees with me, however they offer a get out clause that if the code is identical for each site with the exception of pictures and the odd bit of text content then this may be permissible. Sep 30 '15 at 15:41
  • Interesting 'get out' clause. I guess if the only difference is content and images, the configuration files should be the same so if they're static and monitored (maybe using change detection software) a representative sample could do the trick for scanning. You could always scan the IPs and then cycle through URL address samples monthly or quarterly. Whatever the scheme, it must not compromise whatever security full scanning would offer.
    – AndyMac
    Oct 4 '15 at 4:16
  • Thanks again, our ASV input is simply IP addresses, we do not put anything else in. The platform is outsourced and managed by the hosting provider so do not know what change detection is built into the platform, however this is one of my arguments - I do not believe we can truly know if there has been a change across all 8 servers (albeit only my organisation having the ability to request changes and provide new content via a repository) so including all 200+ would cover this fact. Oct 6 '15 at 13:11
  • The whole responsibility should filter between the different parties involved. For example, you could be providing the infrastructure and address space and perform scanning on that, the hosting provider could perform scanning on their systems and the end merchants could perform scanning on their specific URLs. It could be argued you're providing a compliant base infrastructure but the specifics for scanning higher up are the responsibility of the relevant parties. At an IP level whether 200 or 8 on the same standardised, hardened platform shouldn't make a difference but the URL level might.
    – AndyMac
    Oct 6 '15 at 15:19
1

ASV scan should be per public ip address. It will scan the ip address and also the web application. Although the web application it can be the same in all ip address the services that are available using those ips need to be validated.

Example: The ASV scan needs to confirm that you are only using TCP 80/443 in each ip address. If you are using other TCP ports then you will have to deliver a business justification for that and also if required present compensations measures to make sure your security does not decrease.

2
  • Thanks. How would different services potentially be applied to the various IP addresses ,my understanding is limited to be fair, but I know we simply set an A, MX record etc on our domain registrar account, so we put the IP address in the A record which directs the URL to the load balancer, this will then link through a firewall to one of 8 app servers. Surely the firewall is the system that controls what IP addresses access what services? We will have rule reviews to confirm this is set properly. Then the request evaluates the URL requested and uses one of the 5 aps skinned appropriately. Oct 6 '15 at 13:25
  • It is one audit. They just want to verify from outside that what you are opening is really what should be opened. This validation has to be done from a accredited third party while the firewall validation is internal to your company. You are only required to show the QSA that you have a built in process and you follow it.
    – Hugo
    Oct 7 '15 at 9:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.