Unpacking the Patch Management Process for Operations
March 5, 2020
By Mark Cristiano, Network and Security Services Business Development Manager, Rockwell Automation
The food and beverage industry has seen great momentum when it comes to addressing cyber hygiene – the starting point for industrial control system (ICS) cybersecurity. Where we used to have a lot of conversations about network infrastructure, cybersecurity techniques and strategy are now taking center stage. But how did we get here?
The problem dates back 20-30 years, when the food and beverage industry was rapidly adopting advanced, proprietary technology on the factory floor. Due to the closed and isolated nature of these systems, cybersecurity was not a true concern.
Fast forward to the past 10 years, and the proliferation of ICS and Ethernet-connected equipment has revolutionized productivity, quality, compliance and speed to market. It has also simplified connection of these legacy systems to each other and to new systems. This open, unmodified Ethernet communication brought increased cyber risk and a new concern: legacy system patch management.
A recent Food Protection and Defense Institute report details how this outdated legacy equipment can expose your operation to malicious attacks. Ones that can disrupt business, destroy equipment and compromise worker and product safety. A holistic cybersecurity program has become a business imperative, and the patch management process plays an important role.
You can’t patch what you can’t see
The idea of an asset inventory isn’t new, and you may have already tried this exercise internally, or even enlisted outside help. But to capture everything is no easy task, and many are still working to get it right.
There are two ways to take inventory, and to set the right foundation for your ICS cybersecurity program, you need both.
- – Electronic interrogation tools can scan your network and automatically identify assets, getting you most of the way there.
- – Manual identification will catch the rest, but requires someone to literally walk around, open panels and do a physical survey of what’s out there.
A watch out here is to take both approaches at all of your locations. If only complete at nine of your 10 sites, I can just about guarantee the breach is coming through the one that was overlooked.
Setting a comprehensive patching strategy
Following the inventory, you may be left with a list of thousands of assets to wrap your head around. Luckily, not all assets are created equal. The next step is performing a risk analysis to identify the high priority assets to patch based on their criticality, exposure, age, anticipated risk, etc. Some assets aren’t even on the network, so are they really a risk?
There are two types of patches you’ll need to address:
- 1. Operating system (OS) patching is commonplace for IT, so much so that Microsoft Patch Tuesday has been around for more than 15 years. You’ll have to time plant floor OS patching with scheduled downtime for minimal disruption. Some proactive IT/OT collaboration can take care of this in many instances.
- 2. Application-level patching is a different story. There could be literally hundreds of applications from different vendors with different patches. So it’s incumbent upon you to go find patches on vendor websites, understand the vulnerabilities they protect against and if they are needed or not.
Because each application is configured differently, patching the application layer warrants a very deliberate, consistent testing standard. One conducted in a lab environment prior to implementation on the plant floor where you could run the risk of unintentionally shutting down production.
A systematic approach to patch management
The “fingers crossed” approach is common throughout the food and beverage industry. Not for lack of trying, but for lack of the right resources and specialized expertise. Generally what I see in the field today is reactive. Responding to a high-priority patch notification and accomplished by shutting down production on a weekend as needed.
And the common progression looks like this:
- Operations enlists IT to help manage OT patching.
- IT fills in, but doesn’t have the ICS expertise or resources to manage the unique requirements and constraints.
- So they hire a hybrid IT/OT resource, or more often, outsource to a company like Rockwell Automation or others.
If going the third-party route, seek a partner grounded in operations. One telltale sign is their service level agreement (SLA) response time. Traditional IT providers measure response in hours. But that kind of downtime in consumer goods production can mean millions of dollars lost. SLAs measured in minutes represent an operations-friendly approach.
The ICS cybersecurity end game
Patch management is one step on your way to getting a security operations center (SOC) up and running. An SOC can provide a holistic dashboard view of your security posture, include a disaster recovery strategy and ensure optimal operation of your connected factory.
Additionally, there are solutions available today that are designed for end point protection or “whitelisting.” While these solutions do not entirely eliminate the need for patching, they are an effective solution to protect and buy you time while formulating a patching strategy.
The truth is, there is no silver bullet to effective cybersecurity. That is what defense-in-depth is all about. But with more than the bottom line at risk (think food and employee safety), reaction and a little luck is no longer a viable approach. If you’re looking for a little help kicking off your program, or bringing it to the next level, we’re here to help.
