Data Backup and Recovery in HPC and Big Data

HPC centers are broadly working to modernize their IT infrastructure to meet the quickly arriving needs of tomorrow. Their IT centers face an ongoing challenge to adapt and improve their IT operations to remain flexible and offer the latest performance capabilities. New and different approaches to security, efficiency and performance are needed – and are indeed currently being adopted – to achieve these improvements.

“Of those evaluated, Bacula Enterprise was the only product that worked with HPSS out-of-the-box without vendor development, provided multi-user access (BWeb), had encryption compliant with Federal Information Processing Standards, did not have a capacity-based licensing model, and was available within budget”      NASA

In accordance with its Open Source pedigree, Bacula Enterprise perfectly supports Posix compliant filesystems, helping you to avoid vendor lock-in. In addition, every filesystem that can be mounted to a Linux or Windows host can be used, including parallel and clustered filesystems such as Lustre or Quobyte. POSIX file systems are the most common storage system in use today, providing a wide range of IO functions for applications to use, including byte-level access. However, with the large number of IO functions comes complexity, both for the application and the file system. Bacula helps HPC users to significantly reduce complexity by being file system-agnostic. Here is a non-exhaustive list of filesystems that Bacula Systems customers use:

ext 2|3|4
xfs
ReiserFS
BeeGFS
GlusterFS
GFS
GPFS
Hadoop *
HPSS
Lustre
Quobyte
Swift *

*requires a Bacula Enterprise module (plugin)

As a proven HPC backup and recovery solution of especially high performance, Bacula can handle vast volumes of data with ease. With the increasing need for HPC solutions and further improvements in technology, organizations are turning their attention to areas such as hybrid HPC solutions. Bacula anticipates that technology and innovation improvements in HPC space will increase, especially in specific areas such as Hybrid Cloud, edge compute, container technologies and security approaches. As IT teams work hard to create a balance between on-premises HPC solutions and cloud, Bacula provides a way to protect and recover these entire environments from a single platform.

Read the Bacula whitepaper that covers Backup and Recovery considerations for this sector:

Top 10 Whitepaper Highlights

1. IT environment complexity in the research sector
2. Technical & demanding IT environments
3. Meeting RPO’s and RTO’s
4. The need for especially high levels of security
5. Bare metal recovery
6. The need to de-risk implementation
7. Hybrid cloud in the research sector
8. Stand-alone capabilities and “air-gapping”
9. Container technologies in the research sector
10. How NASA benefits from Bacula
11. Avoiding vendor lock-in

For HPC environments that have petabytes of data to store, tape can still be the best option for long term archival and meeting RPO’s. It’s also very effective regarding retention requirements and media preservation. Today’s LTO-08 and LTO-9 tape drives have impressive specs including transfer rates up to 400 MB/sec. native, and storing up 18 TB of raw and up to 45 TB of compressed capacity. Bacula Enterprise is ideal for tape admins as we never license based on data volume and deliver unique features such as ACSLS support. Bacula supports tape libraries from all of the world’s top manufacturers and provides all tape library management operations. It also supports named user access to ACSLM, tape drive and volume locking in shared ACSLS environment, lock query and management, static tape drive location mapping and dynamic volume location mapping.

Bacula’s especially wide compatibility with HPC technologies is contributing to a positive change in technologies and tools to automate processes and to apply consistent governance across an HPC Federal/National laboratory or research organizations. For example, Bacula offers native integration with technologies such as Hadoop, Docker, Kubernetes, SAP Hana, GPFS, etc. Bacula supports over 33 different operating systems!

As more HPC-driven research and government agencies shift to using Bacula, a pattern of needs clearly emerges from these organizations, and the reasons why the shift takes place. The Bacula Systems white paper ‘Data Backup and Recovery in Federal and National Research Agencies‘ discusses the considerations and advantages of using FIPS 140-2 compliant Bacula Enterprise as a central data backup and recovery system within research organizations, and how it can – and does – facilitate a specific, yet critical part in these organization’s enterprise-wide approach to digital modernization.