Data Backup and Recovery in HPC and Big Data

Home > Enterprise Backup Solutions > HPC and Big Data Backup

Bacula: Unbeatable in HPC and Super Computing Environments.

For an HPC data backup and disaster recovery strategy, you must adapt your IT environment to meet tomorrow’s security standards, integrate with increasingly diverse applications, data, and file types, and meet compliance requirements – and yet reduce costs at the same time. For example, these factors are more critical in the education, research, and defense sectors. There are very few backup vendors that can seamlessly perform these tasks. But Bacula can – with ease. For example, only Bacula can handle billions of files or perform strongly when high throughput is required. Bacula offers exceptionally high levels of security and performance in HPC environments when compared to other vendors.

Read Solution Brief Download HPC & Big Data Whitepaper

“Of those evaluated, Bacula Enterprise was the only product that worked with HPSS out-of-the-box without vendor development, provided multi-user access, had encryption compliant with Federal Information Processing Standards, did not have a capacity-based licensing model, and was available within budget” – NASA

Bacula now offers high performance, native integration with Lustre, GPFS and ZFS filesystems. Contact Bacula now to find out more about its recent announcements that push performance with these systems to a significantly higher level.

With its Open Source pedigree, Bacula Enterprise perfectly supports Posix compliant filesystems, helping you to avoid vendor lock-in. In addition, every filesystem that can be mounted to a Linux or Windows host can be used, including parallel and clustered filesystems such as Lustre or Quobyte. Today, POSIX file systems are the most common storage systems, providing a wide range of IO functions for applications to use, including byte-level access. However, with the large number of IO functions comes complexity, both for the application and the file system. Bacula helps HPC users significantly reduce complexity by being file system-agnostic. Either via Posix or not, here is a non-exhaustive list of filesystems that Bacula Systems customers use:

ext 2|3|4
xfs
ReiserFS
BeeGFS
GlusterFS
GFS
GPFS

Hadoop *
HPSS
Lustre
Quobyte
ZFS
Swift *

*requires a Bacula Enterprise module (plugin)

As a proven HPC backup and recovery solution of exceptionally high performance, Bacula can easily handle vast volumes of data. Bacula also has a number of unique tools available for handing extremely high volumes of data, such as its HPCAccelerator feature (see opposite). Bacula anticipates technological and innovation improvements in the HPC space will increase, especially in Hybrid Cloud, edge compute, container technologies and security approaches. As IT teams work hard to create a balance between on-premises HPC solutions and cloud, Bacula provides a way to protect and recover these entire environments from a single platform.

“Our new backup solution not only had to offer a simple licence model, but also be able to cope with billions of files and petabytes of data. Bacula proved to be a powerful solution that can efficiently backs up large environments.” – University of Ulm

High Levels of Automation for HPC Users

Bacula offers an amazing degree of customization and automation in practically every part of its broad feature-set. Contact Bacula for more details on its high automation levels.

The Broadest Tape Compatibility of any backup vendor

For HPC environments that have petabytes of data to store, tape can still be the best option for long term archival and meeting RPO’s. It’s also very effective regarding retention requirements and media preservation. Today’s LTO-08 and LTO-9 tape drives have impressive specs including transfer rates up to 400 MB/sec. native, and storing up 18 TB of raw and up to 45 TB of compressed capacity. Bacula Enterprise is ideal for tape admins as we never license based on data volume and deliver unique features such as ACSLS support. Bacula supports tape libraries from the world’s top manufacturers and provides all tape library management operations. It also supports named user access to ACSLM, tape drive and volume locking in shared ACSLS environment, lock query and management, static tape drive location mapping and dynamic volume location mapping. Please note: Bacula is an industry leading backup and recovery solution for immutable backup.

Total Fit: Bacula’s Ultra-Broad HPC Compatibility

Bacula’s extensive compatibility with HPC technologies contributes to a positive change in technologies and tools to automate processes and to apply consistent governance across an HPC Federal/National laboratory or research organizations. For example, Bacula offers native integration with technologies such as HPSS, Hadoop, Docker, Kubernetes, SAP Hana, GPFS, etc. Bacula supports over 33 different operating system versions!

As more HPC-driven research and government agencies shift to using Bacula, a pattern of needs clearly emerges from these organizations, and the reasons why the shift takes place. The Bacula Systems white paper ‘Data Backup and Recovery in Federal and National Research Agencies‘ discusses the considerations and advantages of using FIPS 140-3 compliant Bacula Enterprise as a central data backup and recovery system within research organizations, and how it can – and does – facilitate a specific, yet critical part in these organization’s enterprise-wide approach to digital modernization.

HPCAccelerator

This Bacula feature backs up large numbers of files with an optimal speed and efficiency, splitting the workload in a number of workers (concurrent backup processes). Bacula’s HPCAccelerator saves significant time in HPC environments when dealing with very large datasets. It can work with any kind of file system, and the backup is automatically managed by Bacula. This is applicable to any file-server or NAS use-case.

Key Benefits of Bacula’s HPCAccelerator Technology:

Massive Scalability: Dynamically distributes backup workloads across multiple concurrent workers to handle billions of files with ease.
Speed and Efficiency: Significantly reduces backup times by splitting large workloads into smaller, parallelized processes.
Full Automation: Eliminates the need for system administrators to manually partition datasets.
Error Reduction: Automation and adaptive scalability minimize human error and optimize resource use.
Universal Compatibility: Works seamlessly with any file system, file server, or NAS environment, and integrates with Bacula’s incremental and other advanced backup features.
No Capacity-Based Licensing: Unlike most competitors, Bacula never charges by data volume, giving customers complete budget predictability.

Another attraction of Bacula’s automation of this process is its ability to reduce the possibility of human error. It is also fully dynamic, meaning no configuration update process is necessary, regardless of any changes in the structure or size of the target dataset.

This tool can especially help where there are many users, resulting in a complex folder structure and lots of different files. This kind of file structure may literally be being used by everybody (e.g. public/private, internal company users, shared data situations, etc.). More details are available in Bacula’s HPC white paper.

Read Solution Brief Download HPC & Big Data Whitepaper