Patch Advisory Board - feasibility of implementation

Question

I'm working for a public sector organisation (around 6000 staff) with a mixed IT skill set. We're looking at setting up a patch advisory board to assist in server patching.

The plan is to have a team from different areas of the business (Apps, DBs, infrastructure, etc) to review each patch and then approve whether they're installed on any of our 1100 servers. Typically the patches reviewed will be from below.

https://portal.msrc.microsoft.com/en-us/security-guidance

My questions are:

1. How feasible is this approach given the work, effort and time involved to regularly understand dozens of patches and then decide whether to apply them?

2. Given that we don't fully understand our own servers and the patches, reviewing each patch doesn't guarantee success.

3. How many of you use a patch advisory board in your current work place?

As opposed to just applying critical and security patches on dev and test before moving to production

Thanks

Answer 1

We have a similar problem. Thousands of servers, tens of thousands of users, thousands of applications. No one knows anything. We don't have thousands of mirrored dev and test servers: most of our servers are primaries.

Change Advisory Boards are great good practice and a great way to spend hours of effort. They reduce your risk, specifically reduces the risk of "oh dear we didn't expect that I will resign".

What we found was that 99% of security patches just worked. There were some MS ones that borked the OS, but as we typically wait two weeks after release before deploying the patches that's sufficient time for news to leak about a broken patch.

Now our approach is: just apply the security patches. Some high status operational systems get a test first and we do a test on a small sample of servers just to be sure, but our goal and our policy is "everything gets patched".

This is similar to vulnerability management. Rather than employ a team of vulnerability researchers to determine if a vulnerability should be remediated, we just remediate everything except the major categories we have decided to accept: weak SSL cyphers on internal systems springs to mind.

There is a chance that a patch breaks a system but we found we could not completely mitigate that so we patch and be damned and then restore from snapshot. That means our engineers can do productive and useful work and only have to deal with the occasional exception. We can't test all patches on all systems on all possible paths of execution, so even if we do test it's only "does the patch install, is the light still green".