Why Backup QEMU VMs: Methods, Setup, and Best Practices

Why Backup QEMU VMs?

Virtual machines are the backbone of almost any modern IT infrastructure, and QEMU-based VMs are a popular choice in virtual envionments. Creating proper backups of these virtual environments is not just a recommendation, it is typically a required part of any proper business continuity and disaster recovery plan. Properly maintained backups become a company’s safety net when its hardware fails (and there is no such thing as infallible hardware).

Virtual environments have unique advantages over physical hardware in creating efficient and consistent backups.  As for QEMU itself, it is a free and open-source emulator that uses dynamic binary translation to emulate a computer’s processor. QEMU can emulate a variety of computer architectures, operate guest operating systems, and even support many different hardware options. Additionally, QEMU easily operates as a device emulation back-end or hypervisor for VMs, which makes it very appealing to a wide range of users.

QEMU VMs incorporate customized operating systems, critical application data, and valuable configurations. Losing such an environment typically means losing hours or days of setup and configuration work, while also potentially disrupting business operations, customer service operations, and potentially even worse outcomes. As such, this information should be protected, and backups are often seen as one of the most reliable and versatile ways to do so.

Most regulatory compliance frameworks now require backups, including specific retention frameworks. Add that to the fact that backups can also protect information against ransomware attacks, and it is easy to see why this topic is so important.

The investment in proper VM backup strategies pays dividends in many ways: reduced downtime, improved business continuity, and the general peace of mind that comes from knowing that your data is recoverable after virtually any possible disaster. QEMU’s open-ended architecture also makes backup strategies more flexible, making it possible to use both simple file-based approaches and complex incremental solutions. This article explores QEMU backups, reviewing different methods, setup processes, and potential best practices.

Backup Methods for QEMU

There are several different backup types that can be used to safeguard QEMU virtual machines, with each approach having its own benefits and shortcomings. The most effective backup and recovery solution for any specific situation will depend on the company’s performance and security requirements, policies, storage constraints, among other factors, making it unrealistic to identify  one backup solution that is  better  in every situation.

Next, the article explores the primary backup strategies that have been proven effective in QEMU environments.

Full Backup

Full backups should capture all information in a specific location at once, the entire virtual disk with all of its configuration files and other VM information associated with it. In other words, a full backup creates a complete and self-contained replica of a VM, making it easily restorable without requiring any other backup set.

The combination of simplicity and recovery speed is undoubtedly the greatest advantage of full backups. A full backup eliminates the need to piece together several backup components to restore information when disaster strikes: you can just restore the full backup and continue your business tasks. It is a particularly useful method for protecting the most critical VMs in the environment, where the cost of downtime is significantly higher than the cost of storage.

With that being said, full backups do require a significant amount of storage space and network bandwidth to conduct. There is also the risk that information will be duplicated several times over, due to the lack of granularity in full backups, making them even less storage-efficient. As such, environments with limited storage capacity would find full backups impractical as the only strategy, and the same could be said for generally large VMs.

Incremental Backup

Incremental backups can be thought of as the “middle ground” of backup methodology. Once a full backup is complete, all later incremental backups capture only information that has been changed since the last backup (of any type) occurred. That way, backups become both significantly more storage-efficient and exponentially faster than full backups.

QEMU’s incremental backup approach uses ‘block device dirty tracking” via bitmaps to monitor which blocks were changed since the last backup. This mechanism helps minimize the impact of the backup on system performance, while creating a chain of manageable backup files that represent the complete VM state.

With that being said, the restoration process is where the advantages of incremental backups become somewhat less impressive. Each restoration process requires processing both the original full backup and every single incremental file in a specific sequence. Careful attention to managing these chains is necessary to ensure that there is no file corruption or missing links that can compromise the entire backup strategy.

Incremental backups are still fairly popular in most environments in which storage efficiency and smaller backup windows are the priority.

Differential Backup

Differential backups, on the other hand, offer a balance between full and incremental backup methods. Once the initial full backup is created, each subsequent differential operation will capture all changes made since the original backup.

Compared to incremental backups, differential backups offer a much easier restoration process, because only the full backup and the latest differential backup are needed. As a result, restoration processes using differential backups are faster and more predictable, in stark contrast to the slow process of rebuilding long incremental chains. Differential backups are a good compromise for mid-sized environments that need both recovery simplicity and storage efficiency.

The biggest issue with differential backups is simply the passage of time. As time passes since the last full backup, each subsequent differential file grows, sometimes rivaling the original size of a full backup if too much time has passed. As a result, differential backups are typically most effective when there are regular full backups that reset the baseline for differential backups and maintain operational efficiency.

How to Set Up Incremental Backup in QEMU?

Incremental backup implementation in QEMU is particularly interesting, as it is often the preferred method for dealing with this kind of virtualization. Yet again, proper configuration and implementation require a thorough understanding of various underlying mechanisms, something this article covers next. Here, the article covers three important steps of the process: e creating initial backup infrastructure, leveraging libvirt for backup management, and establishing consistent procedures for regular operations in the future.

Creating the Initial Backup Job

Establishing the initial full backup with bitmap tracking is the foundation of any future incremental backup strategy in QEMU. It is a very important step that creates a point all future backups can reference.

The process in question is not particularly difficult, but it can be challenging in some situations. The first step is to create a persistent bitmap to track changed blocks on a virtual disk. This bitmap can be treated as QEMU’s memory, so QEMU knows which disk sectors have been modified since the last backup operation.

An executable command for enabling bitmap (in QEMU monitor) should look like this: block-dirty-bitmap-add drive0 backup-bitmap persistent=on

Once the bitmap has been established, it is time to perform the initial full backup with the running VM in mind. This particular command should only include the bare minimum of configurations:  target location, format, etc.

This example creates a baseline backup file using the qcow2 format, which serves as a starting point for the incremental chain. Storing this base image in a safe environment is paramount, as its corruption can compromise all the incremental backups that use it as a starting point.

Using Libvirt to Manage Backup Operations

Libvirt is an open-source set of libraries and software that provides centralized management for a variety of different hypervisors, including QEMU, Xen, KVM, LXC, VMware, and others. Libvert consists of a daemon, an API, and command line utilities to operate that API.

Libvirt helps elevate QEMU backup management by using a consistent API layer that abstracts the many different complexities in the environment. Libvirt is a powerful toolkit that can enhance hypervisor tasks by providing automation capabilities and a flexible structure, both of which must otherwise be performed through manual command sequences.

The first thing to do after attempting to set up libvirt backups in QEMU is to verify that the current installation supports incremental backup features (all versions above 6.0.0 should support it). The correct command for checking the libvirt version is as follows:

Next, configure the domain XML to include the necessary backup definitions. The current domain XML file can be viewed with:Once the file is extracted, modify the configuration to include backup elements like this:Once the configuration has been changed, the backup operation can be executed with the following command:The ability of Libvirt’s checkpoint functionality to handle coordination across multiple disks, if necessary, can be extremely valuable to users.

Step-by-Step Guide to Issue a New Incremental Backup

Once all the basic configuration processes are complete, regular incremental backups can be executed using the following sequence of commands:

1. To freeze the guest file system (if the guest agent is already configured):

2. To create a new incremental backup while specifying the tracking bitmap:

3. To unfreeze the guest file system to resume normal operations:

4. To reset the change tracking bitmap to prepare for the subsequent backup cycle:

5. To verify completion and documentation of the backup:

6. To test backup integrity on a regular basis to ensure recoverability:

This particular workflow manages to balance efficiency and thoroughness, minimizing the impact on running workloads and also ensuring a reliable backup chain for potential disaster recovery scenarios.

What Are QMP Commands for Incremental Backup?

The QEMU Machine Protocol, often referred to as QMP, offers a JSON-based interface for programmatically monitoring and controlling various QEMU instances. With respect to backup operations specifically, QMP can provide precise control, valuable especially for either automation or integration with custom backup solutions. The following commands can be executed either using the QEMU monitor directly or using scripting to create scheduled operations:

Introduction to Basic QMP Commands

QMP commands use a consistent JSON structure to facilitate tasks such as scripting and automation. Scripting and automation provide fine-grained control over the internal mechanisms of QEMU without direct access to the console interface of a hypervisor.

To enter the QMP mode while QEMU is running, connect to the QEMU monitor socket and initialize the connection in the following manner:

Some of the most valuable commands for backup operations include:

• block-dirty-bitmap-add for change tracking;
• drive-backup for executing backups; and
• transaction for various grouping tasks, etc.

Each of these commands also accepts a number of specific parameters in JSON:

QMP’s structured responses are perfect for parsing programmatic resources. Each command produces a JSON object that represents either success or failure and an abundance of relevant details. Such a structured approach makes error handling of automated backup scripts much more effective, which is an invaluable feature in any production environment.

How to Create a New Incremental Backup Using QMP

Incremental backup creation using QMP is a logical operation sequence that captures only the changed blocks while maintaining data consistency. It also uses bitmap tracking to minimize backup duration and size, the same way it was used in the different examples above.

Establishing a tracking bitmap, if one does not always exist, should be performed only once before a full backup. Here is how it can be done:

Once the bitmap is established, the drive-backup should be used to execute a full backup using the necessary parameters:Any subsequent incremental backups change this sequence in only a minor way, switching full for incremental in backup types and referencing the tracking bitmap created above to capture only changed blocks:

Understanding Backing Images and Bitmaps

The relationship between backing images and dirty bitmaps creates the technical foundation for efficient incremental backups in QEMU. Maintaining clean backup chains is possible only with a proper understanding of these relationships.

Backing images create parent-child relationships between qcow2 files so that each incremental backup can reference its predecessor. Query the backing chain of any qcow2 image with the following QMP command:

The same command can also be used to view existing bitmaps on a specific drive by changing one of the arguments:
Bitmap consistency should be carefully maintained across backup operations to create reliable incremental chains. Once an incremental backup is completed, it is recommended to also clear the bitmap to begin tracking all the changes from scratch for the next potential operation:

A reset operation like this marks the completion of a single backup cycle and prepares the system for executing the following cycle, as well.

Common Issues and Troubleshooting of QEMU Incremental Backups

All the planning in the world may not save QEMU backup operations from encountering any obstacles or issues. Knowing how to diagnose and resolve them efficiently is crucial knowledge that can mean the difference between incurring minor inconveniences and substantial data losses. This section addresses some of the most common challenges administrators face with respect to incremental backup solutions.

“Bitmap not found”

“Bitmap not found” errors usually stem from issues with bitmap persistence. For incremental tracking to be consistent using QEMU, bitmaps must persist across VM reboots. The persistent=on flag should be used when creating each new bitmap, because there is no way to change the existing bitmap’s persistence setting other than recreating it from scratch.

“Permission denied”

Permission errors are fairly common in backup operations, especially in environments with complex security rules. There is a certain test command that can be launched to ensure that the QEMU process has permission to write to your backup destination:

If this test fails, the only solution is to manually adjust permissions or ownership on a backup directory.

“Device is locked”

If certain operations have exclusive locks on the target device, backup operations may fail with the message “device is locked.” Such locks can occur during snapshots or concurrent backup jobs, and the only way to avoid them is to list active backup jobs beforehand to be able to find potential conflicts by hand:

It is also possible to cancel certain operations, when appropriate, with the following command:

Corrupted backup chains

Backup chain corruption is particularly challenging in this context, immediately rendering all subsequent incremental backups unusable. The best recovery approach in situations like these is to create a new full backup and establish a fresh chain to start anew:

Inconsistent application states

the inconsistency can disrupt the backup process and result in incomplete or otherwise damaged backups. In that case, the exact resolution depends on the core of the issue, so there is no single solution for every problem.

For example, if an application was performing write operations during backup, it may result in backups with only partially written data. This can be resolved only by stopping all associated VMs before conducting backup operations and unfreezing them afterwards with these commands:

Disk space exhaustion

Disk space exhaustion can interrupt backup operations, leaving incomplete backup files behind. Such files only consume storage space: they have no recovery value in their incomplete form. Space monitoring is another layer of commands that should be implemented in backup scripts to prevent starting any operations when available space can fall below a certain threshold.

Implementing regular cleanup processes to remove partial backup files should be considered.

“Image not in qcow2 format”

Backup operations can fail with “Image not in qcow2 format” errors, even when the correct format is specified beforehand. Such issues often occur when attempting incremental backups when the base images are stored in an incompatible format.

This can be resolved by first verifying the base image format:

Once the format has been verified, the image in question can be converted into qcow2, while starting a new backup chain, with the following command:
Effective troubleshooting always begins with complex logging. Verbose logging of backup operations ia paramount to capture detailed information when various errors or issues appear:
Such logs prove themselves priceless when diagnosing complex issues that might be practically unsolvable otherwise.

Backup Methods for Running QEMU VMs

There are several noteworthy differences in the two approaches to QEMU backup management that have been covered here.

The first is with the help of QEMU Monitor Commands:  they are performed directly through the QEMU monitor console using text-based syntax and are typically used to perform various tasks manually. While it is true that libvirt offers certain features to assist with automation, its basic idea is still closer to direct QEMU monitor commands in nature.

The second uses QMP, or QEMU Machine Protocol, a system designed for programmatic interactions that can be accessed using a socket connection. It is perfect for scripting, automation, and backup sequencing with all of its JSON-formatted commands and responses.

Their functionality is essentially the same at its core; these are just different interfaces to access the same features of QEMU.

Both of these approaches offer several different ways to create a backup of a running VM in QEMU. Some of these possibilities have already been explored, such as the dirty block tracking, the freezing/thawing capabilities of QEMU’s guest agent, and the checkpoint capability of libvirt.

One alternative that has not yet been mentioned is the external snapshot capability. It is also often considered one of the simplest approaches to working with running VMs by creating a new overlay file toward which all the write operations are redirected, while the original disk image is preserved as-is for the backup process.  A command for using this method looks like this:

Once the entire backup process has been completed, it is important to commit all the changes from the overlay file to the base image in a specific manner:
It should also be noted that some third-party backup solutions offer integration capabilities with QEMU that provide a variety of additional features:  centralized management, compression, deduplication, support for backing up active VMs, etc. They leverage QEMU’s API while adding their own orchestration layers and storage optimization tweaks. To make the topic more clear we can take one such solution and explore its capabilities in  more detail, which is exactly what the article does below with Bacula Enterprise.

All these backup methods have their distinct advantages and production contexts in which they outperform the rest, such as:

• Dirty block tracking with incremental backups: one of the most balanced approaches, offering minimal performance impact and high efficiency; a great option for production environments with backup window limitations and reasonably large VMs.
• Guest agent integration (freezing/thawing): a common option for transaction-heavy applications and database servers that require complete data consistency, even at the cost of brief downtime windows during backups.
• Checkpoint capabilities: provide the most complete recovery, but at the cost of high resource usage, which makes them the preferred option in development environments and critical systems in which additional overhead is justified by preservation of the application state.
• External snapshots: great in environments that need backups with little-to-no setup, making them perfect in small and medium VMs with sufficient tolerance for brief slowdowns.
• Third-party backup solutions: provide the best experience for enterprises with a wealth of VMs and hosts, emphasizing centralized management and advanced features to justify their high licensing costs.

QEMU Backup APIs and Integration Tools

QEMU’s rich API ecosystem offers both developers and administrators deep programmatic access to versatile virtualization capabilities. Such APIs operate as the foundation for backup operations, providing consistent interfaces and abstracting the complexities of managing multiple virtual machine environments.

Block Device Interface is at the heart of QEMU’s backup capabilities. It allows operations for managing virtual disks, including, but not only limited to the backup and snapshot capabilities explained above. This interface can support operations such as bitmap management, blockdev-backup, and drive-backup via both QMP and QEMU monitor. These low-level functions are also perfect for developers creating custom backup solutions, offering granular control over practically every aspect of the backup process.

The libvirt API is another popular option in this context, wrapping QEMU’s native interfaces with a standardized abstraction layer that can even operate across different hypervisors. As mentioned before, libvirt helps simplify backup operations with high-level functions that can handle various underlying details automatically. For example, the virDomainBackupBegin() function can manage all aspects of initiating an incremental backup, from bitmap tracking to temporary snapshots.

As for Python developers, the libvirt-python bindings can be used as a relatively convenient entry point to QEMU’s backup toolset. The bindings provide the complete libvirt API in a Python syntax, making automation scripts much more readable and easier to maintain. Here is how a simple backup script would look in Python:

The standardized nature of these APIs creates a rich ecosystem of third-party backup solutions to expand on QEMU’s existing capabilities. There are many different tools that can leverage these APIs to create feature-rich backup experiences, while simplifying many of the technical complexities this article has reviewed. The remainder of the article explores the essential features of third-party QEMU backup solutions, using Bacula Enterprise to illustrate how a backup solution can work with QEMU’s original feature set.

Essential Features in a QEMU Backup Solution

Certain key capabilities separate robust backup solutions and basic approaches to backup processes. Essential features like the ones mentioned below should ensure that a QEMU backup strategy can remain reliable, efficient, and recoverable across a diverse range of virtualization environments.

Data consistency mechanisms are the most critical feature of any competent backup solution in this context. A backup solution should be easily integrated with QEMU’s guest agent API or offer its own application-aware plugins to ensure database consistency. The ability to coordinate with running applications can help create backups in a clean, recoverable state without any corruption mid-transaction. Advanced solutions for storage-specific use cases that go beyond freeze-thaw cycles should also be considered where applicable, making it possible to manage specific applications’ transaction states on a separate basis.

Efficient storage management is another important point for comprehensive backup solutions, with common features including deduplication, compression, automated retention, and more. Incremental-forever approaches offer minimal backup windows and storage consumption via intelligent change tracking. In this context, automated verification on a regular basis is virtually mandatory, testing backup integrity and recoverability whenever possible to ensure that backups are still viable and complete at all times.

Orchestration and scheduling are both incredibly important for more complex environments, transforming manual backup procedures into reliable, automated processes without the need to create complex scripts in the process. Intelligent resource throttling, dependency management, and flexible scheduling options are all practically expected here. Outside of this basic functionality, comprehensive reporting and alerting mechanisms should be present in any competent backup solution for QEMU, as well as integration with existing monitoring systems and RBAC support for better access control.

All these features become increasingly important as virtual business infrastructure grows both in size and complexity, turning backup from a technical process into a business application with specific governance requirements and defined responsibilities.

How to Backup QEMU with Bacula?

Bacula Enterprise can provide extensive support for QEMU environments using its virtualization module – among other features. Bacula combines the open-source nature of the environment with centralized management, premium support, and fine-grained control over practically every process. Such an incredible combination of parameters makes it a favored solution for large businesses with diverse virtual infrastructure requirements.

Bacula’s configuration for QEMU backups begins with installing the Bacula File Daemon on hypervisor hosts. The daemon should be configured to access your QEMU instances with the help of libvirt, making both full and incremental backups possible without potential instances of data corruption.

A core configuration for these backups is stored in Bacula Director’s configuration file, where users can define backup jobs to target specific VMs:

A configuration like this leverages Bacula’s QEMU plugin to handle all the complexities and nuances of this backup process automatically (including bitmap tracking).

One of Bacula’s strongest features is its use of a catalog-based approach to multi-VM recovery capabilities. Bacula can maintain detailed metadata of each backup and all the relationships between them when necessary. That way, precise point-in-time recovery becomes possible without the need to track backup chains or restoration dependencies manually.

For disaster recovery, Bacula uses its bare-metal recovery capabilities to restore entire hypervisors and all their VM configurations and disk images. Bacula’s comprehensive audit trails and retention enforcements are particularly useful in businesses with strict compliance requirements.

Bacula’s many enterprise features, combined with its open architecture, make it an interesting option for businesses that require robust QEMU backup capabilities capable of scaling from single-server deployments to vast multi-datacenter environments.

Frequently Asked Questions

What are the different methods of backing up a QEMU virtual machine?

QEMU virtual machines have several ways to create backups from them, including full backups, incremental backups, differential backups, and external snapshots.

• Full backups capture the entire VM but require considerable storage space.
• Incremental backups use dirty block tracking to monitor changed blocks efficiently but are difficult to restore.
• Differential backups are the middle ground between the two, but are also not particularly universal in their range of use cases.
• External snapshots redirect write operations to overlay files on a temporary basis while the base image is backed up.

Is it possible to back up a running QEMU virtual machine without downtime?

Yes, QEMU has support for live backups of running VMs using its own mechanisms such as dirty block tracking or external snapshots. For optimal consistency, administrators often use guest agents to briefly freeze the filesystem for critical backups, ensuring app data integrity but making such backups unacceptable for specific business types.

What is the role of the QEMU snapshot feature in backup solutions?

QEMU snapshots create point-in-time captures of the current VM state to serve as a foundation for different backup strategies. The state of internal snapshots is stored within the original file, while external snapshots are redirecting write operations to separate overlay files. Snapshots also help enable various useful features, such as rollback, cloning, migration, and more.

Using a high security backup and recovery solution to protect QEMU environments typically also brings single pane of glass protection to an organizations’ entire IT environment which is like advantageous. It also brings far more monitoring, reporting, compliance, security and convenience features, often required for running medium and large business. We hope this information has been useful to you – you can find out more at www.baculasystems.com.