Application Consistent Backups in NetBackup OpenStack 9.0

Description

Create application consistent backups using QEMU agent in the guest

Introduction

NetBackup for OpenStack by default creates crash consistent backups. Crash consistent backups are sufficient for majority of the applications. 
Though enterprise databases such as Oracle, MS SQL have resiliency build into them that keep the databases in consistent state during system crashes, for better durability it is recommended to take application consistent backups. 
Application consistent backups involve four main steps:

1. Flush applications buffers.
2. Quiesce the application so the database on the disk remain in consistent state.
3. Take snapshot of the disks that make up the VM and database.
4. Thaw the application so the database on the disk continue to receive updates.

NetBackup for OpenStack already uses Cinder and Nova snapshots to perform VM snapshots. Other steps need to be performed within VM/Guest and require some application specific logic to be executed. 
NetBackup for OpenStack uses QEMU agent in the guest to orchestrate application consistent process by invoke freeze/thaw actions on the QEMU guest agent in the the VM. Guests can register freeze and thaw hooks that invokes application specific APIs/CLIs to per flush application buffers, quiesce the application and then thaw the application.

Nova support for QEMU Guest Agent

Nova usually does not create the Unix pipe such as org.qemu.guest_agent.0 when an instance is created. As mentioned above, Unix pipe is needed for libvirt to communicate with guest agent in the instance. In order for Nova to create the Unix pipe, you need to tell Nova that the instance need QEMU guest agent. There are two ways to specify QEMU guest agent enablement to Nova.

Creating Glance image with hw_qemu_guest_agent property

When the OS image is uploaded to glance repository, add additional property hw_qemu_guest_agent=yes as shown below:

​$ glance image-create --name cirros --disk-format raw --container-format bare\
 --file cirros-0.3.3-x86_64-disk.raw --is-public True --property hw_qemu_guest_agent=yes\
 --progress

You can also update the image and add the property and subsequently when Nova creates a new instance from this image, it creates the Unix pipe. 
The Unix pipe appears as a character device in the guest and QEMU guest agent listens for commands from hypervisor.

Adding hw_qemu_guest_agent property on the instance

Sometimes, adding property on the glance image is not sufficient. You may have migrated the instance from a different cloud or you may have created the instance from a cinder boot volume in which case you can add the property on the nova instance and hard reboot the instance. During hard reboot, Nova creates the Unix pipe and exposes it as character device in the guest.

nova meta <server-guid> set image_hw_qemu_guest_agent=yes
nova reboot --hard --poll <server-guid>

Installing QEMU guest agent in an Instance

ubuntu@agent:~$ sudo apt-get install -y qemu-guest-agent
ubuntu@agent:~$ sudo mkdir /var/log/qemu-agent
ubuntu@agent:~$ sudo tee /etc/default/qemu-guest-agent > /dev/null <<EOF
DAEMON_ARGS="--logfile /var/log/qemu-agent/org.qemu.guest_agent.0.log --fsfreeze-hook --verbose"
EOF
ubuntu@agent:~$ sudo service qemu-guest-agent restart
 * Restarting QEMU Guest Agent qemu-qa
   ...done
ubuntu@agent:~$ sudo ls /var/log/qemu-agent/
org.qemu.guest_agent.0.log

For Red Hat Enterprise Linux 8.x and 9.x:

When selinux is set as enforcing then qemu-agent-command 'guest-fsfreeze-freeze' fails.
error: internal error: unable to execute QEMU agent command 'guest-fsfreeze-freeze': failed to open /volume_mount_point : Permission denied

To resolve this issue:
$ setsebool -P virt_qemu_ga_read_nonsecurity_files on
$ getsebool virt_qemu_ga_read_nonsecurity_files
virt_qemu_ga_read_nonsecurity_files --> on
after executing above commands Freeze and thaw command returns : {"return":3}

Applicable to:

OpenStack Distribution: RedHat 16, 16.1, 16.2, 17.1
Components: Datamover
NBOSVM Version: 9.0.0.1 onwards