Industrial Cybersecurity: How to Deploy Cyber-Resilient Backups

A mechanical failure or a cyberattack costs an industrial operation far more than it costs an ordinary enterprise IT operation. Unforeseen downtime for large-scale manufacturers now averages $260,000 per hour. Yet, the figure swings drastically from industry to industry, with the automotive industry sitting at the far end of the spectrum.

It’s estimated that downtime for an automobile manufacturer, as a result of mechanical failures or cyber threats such as ransomware, reaches a colossal loss of $2.3 million every hour. According to IBM X-Force, manufacturing accounts for 27.7% of cyberattacks, which makes it the most targeted industry for cyberattacks for five consecutive years. And given that downtime costs a manufacturer such an astronomical loss, attackers know that in the event of a breach, the victim is likely to pay the ransom quickly to get back online

To protect an industrial operation and recover from a cyberattack, a manufacturer needs to deploy a cyber-resilient, multi-component modular software solution that’s compatible with both OT and IT systems for immediate point-in-time recovery.

In this article we cover how IT and OT systems are protected, the common challenges industrial systems face, and how they can effectively defend their systems from cyber threats and reduce downtime in the event of a crash.

What Are Industrial Control Systems (ICS)?

Put simply, an industrial control system (ICS) is the underlying hardware and software that operates physical equipment in factories and plants. It comprises four core parts: programmable logic controllers (PLCs), which execute the commands that run a production line; SCADA systems for monitoring and controlling processes in a plant; human-machine interfaces (HMIs) that let operators view and adjust those processes; and sensors that are used to report temperature, pressure, flow, and other readings back to the system. ICS is a specific component within OT (operational technology), whose protection differs greatly from that of an IT-operated system.

What Is the Difference Between OT and IT Security?

The manufacturer runs both IT and OT systems. The former is the corporate side, with its business systems. The plant floor, with its controllers and production systems, is the latter, and it can’t be secured the same way as IT. The standard response in IT environments is usually to shut the affected system down and contain it.

By comparison, OT systems can’t be halted, because the physical process a system runs becomes dangerous if stopped without immediate precautionary measures. In other words, the move that contains an attack in an office can cause a catastrophic accident on a plant floor.

The second major problem with OT environments is that much of the equipment on a plant floor runs on old operating systems that no longer receive security patches, which leaves known vulnerabilities wide open.

In factories, controllers have to be restored to a known-good configuration and verified before they drive physical equipment. Every hour of downtime loses revenue and raises safety risk, and because those costs continuously accumulate the longer the outage runs, the manufacturer’s ability to recover quickly is paramount in containing further damage.

In order to safeguard an active industrial system, industries should deploy a cyber-resilient backup and recovery solution handling both IT and OT recovery that’s aligned with the IEC 62443 industrial security framework.

What is IEC 62443 and why is it important in securing industrial systems?

For industrial cybersecurity, IEC 62443 is a widely adopted standard that defines how operational technology (OT) and Industrial Automation and Control Systems (IACS) must be secured. This framework is particularly prevalent in sectors the likes of energy, manufacturing, transport, water, and healthcare.

The Zones and Conduits Model for Protecting Industrial Systems

The philosophy of IEC 62443 revolves around the split of the operational system into zones and conduits, each assigned its own security level ranging from 1 to 4. Zones are the physical groupings of industrial assets such as machines, control systems that, more or less, share the same level of security requirements. Conduits, on the other hand, are the strictly controlled communication pathways that connect these zones.

As a result of this type of segmentation, attackers can’t move laterally into critical zones in the event of a breach. As to the question whether “wouldn’t it be better to apply the same military-grade SL4 across all for ultimate protection,” the answer is no. Should all components be secured with the same security measures, it will bring forth unwanted over and under-engineering of low-consequence zones and life-safety zones, which entails a plethora of complications.

For instance, applying military-grade Security Level 4 controls to the office printers would waste immense resources and paralyze daily tasks, whereas applying basic Security Level 1 defences to the turbine valves would invite a catastrophic physical disaster.

Security Level System (SL1-SL4)

IEC 62443 defines four security levels, each matched to the capability of the attacker you are defending against. Instead of applying one level across the whole plant, manufacturers assess the risk of each zone and assign it a target security level.

– SL1 protects against human error and untargeted low-impact malware (e.g., a worker plugging in an infected personal USB drive).

– SL2 safeguards against low-cost attacks by opportunistic hackers using public tools to exploit basic vulnerabilities

– SL3 is used for protection against sophisticated cyber threats by organised cybercriminals who have specialised knowledge of industrial systems (e.g., a professional ransomware group actively trying to hijack a specific plant’s PLCs).

– SL4 protects against massively funded nation-state military/intelligence agencies that use advanced and unlimited resources to cause physical destruction

The security of industrial systems can be provided with a cyber-resilient backup and recovery solution that’s capable of adapting its protection to each zone’s required security level. Bacula Enterprise’s backup and recovery is carried out by five independent components (the Director, File Daemon, Storage Daemon, Catalog, and Console), which can be distributed and configured according to a plant’s zone structure to provide security across the entire plant.

Real-World Cyber Attacks on Industrial Control Systems

The following three real-world cyberattacks illustrate what happens when an OT system lacks proper safeguards like the ones described above, and the catastrophic consequences that can follow.

The Ukraine Power Grid Attack (Remote Exploitation)

On December 23, 2015, three Ukrainian regional distribution companies were targeted by cybercriminals in an attack that resulted in a pervasive power outage for 225,000 residents.

The adversary hijacked VPN sessions lacking two-factor authentication, which gave them a direct pathway to the companies’ SCADA system. Once in, they manipulated the control systems of the power grid and opened breakers at over fifty substations by hand.

Following this, the attackers deployed malware into the recovery path and wiped files across servers and workstations, erasing the backup and configuration files needed for system restoration. The operations centre’s own uninterruptible power supply was switched off, and the restore path was decimated.

Because the recovery data wasn’t stored on an air-gapped system or on immutable storage, the utilities couldn’t recover on their own, and engineers had to be sent into the field to close breakers by hand.

The Colonial Pipeline Attack (Credential Compromise)

Colonial Pipeline carries fuel through 260 delivery points across thirteen states, and supplies 45% of the fuel used on the East Coast. It’s also the largest refined-products pipeline in the United States.

On May 7, 2021 Colonial’s security system fell victim to a ransomware attack carried out by DarkSide. By gaining access to the leaked password, the attackers accessed a legacy VPN account that had fallen out of use, which gave them a foothold into Colonial’s systems.

The attackers encrypted IT systems and stole roughly 100GB of data. Although they never reached the pipeline controls, Colonial shut the pipeline down on the same day and authorised a $4.4 million ransom payment within hours. Fuel deliveries halted for six consecutive days, and, in turn, gas stations across the Southeast ran dry.

Colonial shut down due to its inability to prove its control systems were clean as the ransomware had hit the office network. What cost Colonial six days wasn’t knowing what the attackers had tampered with, and having no fast way to rebuild the control systems if they had touched them.

To survive the aforementioned incidents, a cyber-resilient backup platform splits its architecture into five independent components to prevent a plant-floor breach from reaching the backup vault by strictly isolating data paths and security roles.

This absolute isolation keeps your audit logs clean so you can see exactly what an attacker touched, while bare-metal recovery allows you to quickly rebuild compromised control systems from a known-good state.

The SolarWinds Breach (Supply Chain Risk)

Roughly 18,000 organizations installed a backdoor malware called SUNBURST into their own networks by applying a routine software update from a vendor they trusted. Among them were the US Departments of Homeland Security, State, Energy, and the Treasury.

Attackers reached SolarWinds in August 2019 and injected malicious code into the build process for Orion (its network monitoring platform). The customers received trojanised updates as legitimate, signed software. The attacks were later attributed to APT29, which is Russia’s Foreign Intelligence Service. FireEye later found the backdoor in December 2020 while investigating a breach of its own red team tools.

Given that Orion is IT software that sits in the data centre, not on the plant floor, you wouldn’t expect it to threaten OT. Yet it does, because the office network and the plant network are connected, and a compromise on one side is able to cross to the other without much hassle.

Challenges with Manufacturing Cybersecurity

Unlike a data centre, a manufacturing plant is many environments all at once, and is a combination of decades-old PLCs, modern cloud-connected IIoT devices, historian databases, and enterprise systems, each with its own patch cycle and downtime tolerance.

This type of heterogeneity is the reason why OT systems are more challenging to protect compared to that of IT, as no single “uniform” backup approach works for both a legacy controller running an unsupported OS and a live historian that can’t be quiesced.

Recovery Across a 30-Year Equipment Lifecycle

Industrial equipment is often two decades older than anything you’d find in a corporate server room, and it stays in service long after IT hardware would’ve been replaced. For instance, a turbine, a press, or a PLC can run for twenty or thirty years, whereas a laptop gets replaced every three to five.

For decades, this wasn’t a security problem, because the plant network was air-gapped, and cut off from any outside network an attacker could use to reach it. However, with Industry 4.0 ending that, modern plants operate across the entire Purdue Enterprise Reference Architecture (Levels L0 to L5), according to which a single backup strategy must adapt to completely different Recovery Point Objective (RPO) and Recovery Time Objective (RTO) requirements depending on the layer.

Upon connecting to corporate IT and the cloud, manufacturers risk exposing the decades-old controllers of the plant to threats the designer hadn’t anticipated.

A plant will need to restore data long after the backup software that wrote it is gone. By utilising a cyber-resilient backup and recovery solution with a documented, self-readable volume format, data stays recoverable on its own, without depending on the original backup system remaining intact. A volume written today can easily be read back decades later, even if the underlying backup deployment has long since been retired.

Backing Up a Live Historian

A data historian is a specialised software that continuously captures and stores time-series data from plant equipment that run on PLCs, SCADA systems.

For regulated industries like pharmaceuticals, databases like AVEVA PI and GE Proficy hold the legal audit trails. In the event of a cyberattack wiping out or encrypting these records, a company loses its proof of compliance. Consequently, a ransomware attack on the plant’s data storage can put a company’s ability to legally ship products at risk, even without ever touching the machines themselves.

That being said, the shortfall with a live historian is that it’s an always-writing database, and write speed is its whole design constraint. As a file-level copy of an active database results in corrupted data, backing up a live historian while it’s running is impossible. In the past, operators would halt the system entirely that led to regulatory and compliance failures.

An advanced modular backup software like Bacula Systems can back up a live historian using native tools such as RMAN for Oracle, or utilising its specialised bpip plugin, which stream live data dumps to storage.

Manufacturing Cybersecurity, Backup, and Recovery with Bacula Enterprise

Bacula Enterprise is a cyber-resilient backup and recovery solution, which is built for security-critical environments and is trusted by organizations such as the U.S. Air Force, NASA, the U.S. Navy, and other conglomerates.

It’s a modular software with five independent components such as the Director, File Daemon, Storage Daemon, catalog, and console. It follows a segmented workflow to prevent lateral movement in the event of a breach, which keeps life-critical infrastructures secure and untouched.

As opposed to the single all-in-one design of most legacy backup tools, each of these five components runs as a separate network service. In turn, the compromise of one component can’t bring the entire operation down.

With its global endpoint deduplication and compression technologies that reduce storage costs, Bacula Enterprise uses a flat, transparent subscription-based licencing model that completely excludes capacity-based pricing.

How Bacula Meets IEC 62443 Security Levels

Bacula aligns with the IEC 62443 for industrial cybersecurity. Per the IEC62443 framework, you can assign different Security Levels (SL) for different zones in your plant, with S1 being used for basic authentication and SL4 for military-grade data protection.

• SL1.  Bacula’s communications between the five components run over TLS by default with automatic pre-shared-key handshakes between the Director, File Daemon, and Storage Daemon. As a result, backup traffic’s never sent in the clear (even at baseline).
• SL2. Each operator is restricted to specific actions, to prevent someone who can run a backup from deleting one, or altering the catalog. Data is encrypted per client at AES 128, 192, or 256, and can be encrypted at rest on the storage target.
• SL3. Multi-factor authentication runs through a TOTP plugin compatible with RFC 6238, which requires a second factor from a phone authenticator for console access. Air-gapped storage applies here as well. Tape ejected from the library or written to offline media can’t be reached by anything on the network.
• SL4. Bacula runs FIPS 140-3 validated encryption through its OpenSSL-FIPS cryptographic module, which allows for highly secure, audited workflows. It also authenticates between daemons with SCRAM-SHA-256, and writes tamper-proof audit logs with chain-of-custody for every backup event. Its Defence and Military profile supports compartmentalised backup in classified zones.

Bacula’s Architecture Keeps Backups Intact

In both the Ukraine grid attack and the Colonial ransomware, the damage spread as far as the attacker’s access reached. These cyberattacks wiped restoration files in one and encrypted every reachable system in the other.

Bacula’s File Daemon runs on the protected machine, which is usually the first component an attacker compromises. It has no interface to the backup storage repository. An attacker who’s fully in control of a plant workstation can’t use it to reach, or encrypt the backups, because the path simply doesn’t exist.

It’s a significant advantage over other backup software for manufacturers. In OT, the client machines are the first to be compromised, and because those machines can’tt reach the backups, the recovery data remains intact.

• Cyber Resilience – Bacula makes use of the 0-3-2-1 rule through its Copy and Migration Jobs, with three copies, two media types, one offsite, zero errors. A plant can write its primary backup to on-premise disk, then copy it automatically to cloud object storage, and later push a third copy to air-gapped tape, with no manual step to forget.
• Immutable storage – WORM-backed volumes and append-only disk lock recovery points against modification once written, although software-controlled immutability is less absolute than tape in a vault.
• Data-poisoning detection – BGuardian analyses every backup job statistically and detects deviations. A Full backup that suddenly protects no data, or a job whose size and file count fall outside the expected range, gets caught, because that pattern is the signature of ransomware encrypting a source before the backup runs. BGuardian also audits the environment’s own configuration and generates hardening reports to surface weaknesses before an attacker finds them.

 Recovery Built for Industrial Systems

Intact backups are useless if you can’t restore them fast, and to the right place. Bacula’s built for both scenarios: whether a plant needs a full system rebuilt from scratch or a single point-in-time restore of critical data.

• Bare-metal recovery –  Bacula can create a complete (from the OS up) rebuild of either a controller or an engineering workstation on both Linux and Windows if either is encrypted or destroyed. This capability turns what’s typically a multi-day shutdown into a revert that can take as little as a couple of hours.
• Restore to a clean machine –  Bacula can restore to any given client. A plant recovering from a ransomware attack can restore to an isolated system while infected machines are still under investigation. Restoration orders can be changed as well, where important process data is brought back to production and the infected systems are sent to a lab for forensic analysis.
• Point-in-time Recovery (PITR) – Bacula captures system and data changes in real time as they happen. This allows operators to perform granular restores to any given millisecond before an incident, which in turn reduces potential data loss to seconds instead of hours.
• One platform across the whole plant – Bacula runs on more than 34 operating systems and protects physical machines, VMware, Hyper-V, Proxmox, Nutanix, containers, Kubernetes, and the databases historians run on, all under one Director. High-speed Global Endpoint Deduplication cuts the storage and bandwidth that heavy telemetry and historian data would otherwise consume.

Scalability and Licencing

• Licencing – In addition, Bacula charges based on the size of the environment and not the size of the data. Therefore, a plant’s backup expenses will not grow each time a new sensor is added. Modern factories produce ever increasing volumes of data, but the operation won’t have to pay astronomical amounts for it. That design also adjusts to environments with billions of files.
• High-Performance Telemetry Management – To easily manage massive streams of time-series data and historian logs that’s generated on a modern plant floor, Bacula employs high-performance Global Endpoint Deduplication and advanced compression technologies. This drastically minimises physical storage infrastructure costs and reduces overall network load without sacrificing critical backup speeds.
• Edge Computing and Smart Factory Scalability – As modern smart factories increasingly process data at the edge, Bacula excels at managing backups for highly distributed environments and decentralised edge devices.

Frequently Asked Questions

What Should an OT Security Strategy Include?

Effective OT security adheres to the standards of  IEC 62443 security with four control levels, and splits network segmentation into zones and conduits. As a result, a breach in one area is able to spread to critical controllers.

Aren’t Factory Systems Air-Gapped and Closed Off From Outside Access?

Rarely, anymore. True air-gapping, where a network is physically isolated with no connection to any outside system, has, nowadays, become the exception. Modern plants connect their OT systems to corporate IT, cloud analytics platforms, and remote-access tools because it makes operations more efficient. But each of those connections also removes a piece of the isolation that used to protect them.

What’s the Difference Between IT Security and OT Security?

IT security and OT security differ in many ways. IT security is mostly focused on protecting the confidentiality of digital assets. In OT, production process integrity and physical safety take precedence over confidentiality. Protecting OT environments is often more difficult and requires a cyber-resilient backup and recovery solution that is able to back up the ever-running live historian consistently.

How Often Should Industrial Systems Be Backed Up?

As often as each system’s tolerance for data loss demands, which means different schedules for different parts of the plant. There’s no single answer, because a plant is many environments with different recovery requirements. Unlike enterprise systems where the daily backup is the standard, the backup frequency for industrial systems varies depending on how much data loss each system can tolerate.

Can Ransomware Affect Physical Equipment As Well?

Yes, in two ways. Firstly, ransomware or malware is able to take a production line offline through encryption or destruction of the system controls. More commonly, an attack on the IT side automatically triggers a precautionary shutdown of OT, even when the malware never reaches the controllers. That emergency shutdown, done outside the equipment’s normal safe-stop sequence, can itself damage machinery built to run continuously.