17

I have recently moved to the manufacturing sector to take care of security of systems/products, specifically operational technology (OT) products. Based on a recent US CISA advisory, I had to apply a patch to multiple units of the same series/product from a particular vendor. The products are used in the assembly line for automating certain jobs.

How do I make sure that the new update will not brick the products installed in the assembly line?

Here, what are the best practices to do the patching for these kinds of products which will not disrupt the line? The products could be patched only during the shift changes (3 shifts are going on).

"Some industrial sectors require 99.999% or greater ICS uptime. This requirement relates to 5 minutes and 35 seconds or less allowable downtime per year for any reason, making unscheduled patching out of the question."

This is true to a large extent as unavailability for even a single day is not acceptable.

Also, as the products are very costly, I can't afford to have a standalone unit to check whether the new software update is not bricking the unit and fully compatible.

In this case, should I be asking the supplier of the product or the manufacturer of the product to provide evidences, that the new update is safe to install?

Improve this question
4
  • 2
    Is the product designed to be used in this kind of environment? If so, I would hope that the software components that need patching would be on a separate, swappable daughterboard or similar component, so you could have a spare daughterboard but not duplicate other equipment. At some point this gets into fitness-to-purpose (or if the manufacturer's design intent is for each customer to have a spare on-site, then perhaps it comes down to the individual customer being too cheap to follow relevant best practices). Regardless, this is something the designer of the product should be considering.
    – Charles Duffy
    Commented Mar 14, 2022 at 19:09
  • 1
    (In a past life, I was lead engineer for a piece of expensive equipment; we had the operating software live as a read-only image on a dedicated two-drive RAID-1 array, and by the end of my tenure one absolutely could swap out that array for two drives with a newer version of the software).
    – Charles Duffy
    Commented Mar 14, 2022 at 19:18
  • 1
    @CharlesDuffy I do OT security assessments including patch assessments. It's not always that simple or that easy.
    – schroeder ♦
    Commented Mar 14, 2022 at 20:56
  • 2
    Any reason not to contact the manufacturer of this very costly product?
    – Pablo H
    Commented Mar 15, 2022 at 13:42

3 Answers 3

Reset to default
14

OT is a different world.

First, what schroeder said. You want to contact the vendor and discuss this with them.

You also want to check with plant management regarding any certifications, Health-and-Safety inspections and other stuff that might become invalid if anything in the system is changed.

You also don't apply patches just because they're there. We do that in IT, because it's usually better to be patched than not. That is not necessarily true in the OT world. In fact, you'll find plenty of outdated systems where the risk analysis came to the conclusion that that's better or that other mitigation measures cover the risk sufficiently.

So no, you never have to apply some patch. You always choose to do so, based on the vendor recommendations, security/risk analysis and operational requirements.

So, in short, here's what I'd recommend (I am currently acting CISO of a manufacturing plant):

  1. be clear why you decide to apply that patch, if there are alternatives (such as other mitigations) to doing so and why you discarded those. I strongly recommend having that in writing.
  2. contact the vendor and inform them that you see the need to apply a patch and schedule a discussion with them. They are the ones who should check the patch works on their systems and guide you through the patching process itself.
  3. contact plant management and inform them that you want to apply a patch, and that according to the vendor this will cause a downtime of X. Get their approval. Also list the risk that the downtime could be longer than anticipated and have them sign off on that risk.
  4. once everyone who is in any way involved is on board, schedule the maintenance, do all the paperwork, let the vendor come in to handle or at least assist in the actual process, and have a rollback/restore plan ready, just in case.
Improve this answer
1
  • 3
    OP stated "Based on the recent US CISA advisory..." but didn't specify which one. It's entirely possible the advisory only applies in specific conditions which may not be met, always important to read the details carefully to figure out the actual risk.
    – barbecue
    Commented Mar 16, 2022 at 21:12
35

Don't touch anything on the line without the vendor providing direct oversight, or even better, doing the work themselves. This should be performed under a support/maintenance contract.

If you cannot test the patch, then you need to transfer the liability to the vendor and be protected by support contracts.

Improve this answer
7
  • 1
    It makes sense to use the contractual agreement if it covers the AMC(Annual Maintenance Contract) and directly work with the vendor to do the patching. What do we do in case of no such support/contract is available?
    – Baranikumar Venkatesan
    Commented Mar 14, 2022 at 11:06
  • 16
    Well, with the addition you've made to the question, you now see the predicament companies put themselves in. They need nearly 100% uptime, but do not provide the resources to be as resilient as they need to be able to do something like patching. They either test the patch on a vendor's demo equipment, get the vendor to do it, or patch and pray. Trying to hope for resilience doesn't work.
    – schroeder ♦
    Commented Mar 14, 2022 at 13:48
  • Can you take one device/line offline and test the patched device there? As a scheduled maintenance period.
    – Rodrigo Murillo
    Commented Mar 14, 2022 at 15:08
  • @RodrigoMurillo from the description from the OP, no. 5-nines uptime means no "offline" time.
    – schroeder ♦
    Commented Mar 14, 2022 at 15:30
  • 12
    Yes, I saw that. Keep in mind that depending on the SLA and what parties agreed to, business objectives, etc, uptime guarantees sometimes do not include scheduled outages or maintenance. The parties might agree to 8 hours of maintenance per month for example, and still be obligated to meet the 5-nines outside of that period.
    – Rodrigo Murillo
    Commented Mar 14, 2022 at 16:10
10

One system I worked on had a simple solution. To guarantee 5 nines, redundant hardware was used. For patching purposes, a fail-over could be initiated manually. One half of the system was patched, another fail-over was initiated, and then the second half was patched. Patches could be reverted in case of failure, and the impact of the temporarily non-redundant system failing was acceptable (<5 minutes).

Of course, this could be generalized to triple-redundant systems so you can fail-over even during patching. Reliability comes at a price.

Improve this answer
2
  • 5
    If you are lucky enough to be able to have redundant equipment. Many OT environments are physically built around the machinery and there is no room for redundancies.
    – schroeder ♦
    Commented Mar 15, 2022 at 16:41
  • @schroeder: IT hardware usually isn't the biggest thing of the shop floor. Having redundant machinery would be a pain, though, and if the IT hardware is physically part of the machinery then the redundancy is determined by the manufacturer.
    – MSalters
    Commented Mar 15, 2022 at 16:54

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .