Deduplication Volumes. Variable Block Deduplication.

Save disk space and money with the unique feature from Bacula Systems – Deduplication Volumes.

• No other backup software stores data this way (patent pending technology)
• Bacula Systems helps you overcome your scaling challenges
• Raising the record size limit brings a major positive impact to your storage costs

Deduplication refers to any of a number of methods for reducing the storage requirements of a dataset through the elimination of redundant pieces without rendering the data unusable. Unlike compression, each redundant piece of data receives a unique identifier that is used to reference it within the dataset and a virtually unlimited number of references can be created for the same piece of data.

It is popular in applications that inherently produce many copies of the same data with each copy differing only slightly from the others, or even not at all. There are many storage systems and applications on the market today which implement deduplication. All can be classified into one of two deduplication types depending on how they store their data:

• Fixed Block
  Deduplication takes places in units of a fixed size (typically 4kB -128kB). Data must be aligned on block boundaries to deduplicate
• Variable Block
  Deduplication takes place in variable-length units anywhere from a few bytes to many gigabytes in size. Block boundaries do not exist.

Deduplicable filesystems use fixed block deduplication.  The optimal unit of deduplication is the record size and it varies depending on the filesystems. For example, 128kB by default for ZFS, 4kB for NetApp. Others available on request.

The Traditional Way

Traditional  backup  programs  were  designed  to  work  with  tapes.  When  they  write  to  disks, they use the same format only writing to a container file instead of a tape. The Unix program tar is an example of this and so is Bacula’s traditional volume format.  Files are interspersed with  metadata  and  written  one  after  the  other.   File  boundaries  do  not  align  with  block boundaries as they do on the filesystem.

For this reason, backup data does not typically deduplicate well on fixed-block systems.

It is the main reason why variable  block  deduplication was invented  and  why  there  is  a  large market for deduplicating backup appliances.

Storage without deduplication

The new era. Bacula Enterprise Deduplication Volumes.

Deduplication Volumes store data on disks by aligning file boundaries to the block boundary of  the  underlying  filesystem.   Metadata,  which  does  not  align,  is  separated  into  a  special metadata  volume.  Within  the  data  volume,  the  space  between  the  end  of  one  file  and  the start of the next block boundary is left empty. Since every file begins on a block boundary, redundant data within files will deduplicate well using ZFS’s fixed block deduplication.  This type of file is known as a sparse or holey file.

Storage with Deduplication Volumes

What to deduplicate?

The following data types deduplicate well:

• Files that change constantly but are only appended to large log files
• Large files that change daily but only in small amounts
• Monolithic Databases
• Some types of email boxes
• Identical files that appear in backups from many clients
• Operating system data from virtual machines
• Email attachments with multiple recipients

Deduplication Volumes are limited only by ZFS itself:

• Data will not dedupe across zpools
• Deduplicated metadata is stored in the ARC/L2ARC
• Only 1/4 of the ARC/L2ARC is reserved for metadata
• Large dedupe repositories will require a large ARC/L2ARC

• The amount of data to deduplicate
• The block size used for deduplication

How to size?

Accurate sizing is difficult in practice.  Oversizing and using conservative estimates is recommended. To help in sizing your infrastructure for deduplication, Bacula Systems provides an online deduplication sizing calculator.

Sizing Recommendations

• Install as much RAM as possible (ARC)
• Use only SSDs for your L2ARC
• Create a much larger L2ARC than you think you need

How sizing impacts your costs?

Current  storage  pricing  trends  bode  well  for  ZFS  deduplication.  Solid  State  Drive  (SSD) performance continues to increase and prices have come down significantly in the past years making  large  L2ARCs  economically  feasible.  The  combination of fast I/O processor (IOP) performance and large capacity is essential to maintain performance as  the amount of data stored in the filesystem increases.

Deduplication Volumes is supported with:

• Nexenta Systems OpenStorage Appliances
• NetApp Data ONTAP 8.0.1 and higher
• Oracle / Sun ZFS Storage Appliances
• White Bear Solutions WBSAirback Appliances
• ZFS on Linux (64-bit only)

Download free trial
Free backup infrastructure assessment