|
| 1 | +.. Licensed to the Apache Software Foundation (ASF) under one |
| 2 | + or more contributor license agreements. See the NOTICE file |
| 3 | + distributed with this work for additional information# |
| 4 | + regarding copyright ownership. The ASF licenses this file |
| 5 | + to you under the Apache License, Version 2.0 (the |
| 6 | + "License"); you may not use this file except in compliance |
| 7 | + with the License. You may obtain a copy of the License at |
| 8 | + http://www.apache.org/licenses/LICENSE-2.0 |
| 9 | + Unless required by applicable law or agreed to in writing, |
| 10 | + software distributed under the License is distributed on an |
| 11 | + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY |
| 12 | + KIND, either express or implied. See the License for the |
| 13 | + specific language governing permissions and limitations |
| 14 | + under the License. |
| 15 | +
|
| 16 | +.. _KBOSS Backup and Recovery Plugin: |
| 17 | + |
| 18 | +KBOSS Backup and Recovery Plugin |
| 19 | +================================ |
| 20 | + |
| 21 | +About the KBOSS Backup and Recovery Plugin |
| 22 | +------------------------------------------ |
| 23 | + |
| 24 | +The KVM Backup on Secondary Storage Backup and Recovery Plugin provider was designed to provide a complete B&R experience for KVM. It uses already validated methods |
| 25 | +to create incremental backups of VMs. Its main characteristics are: |
| 26 | + - Full or incremental backups, configured by the user. |
| 27 | + - Support for backup compression. |
| 28 | + - Support for backup validation. |
| 29 | + - Support for "quick restore", inspired by Veeam's Instant Recovery. |
| 30 | + - Incremental disk-only VM snapshot compatible (see :ref:`Disk-only-File-based-Storage-Instance-Snapshots-on-KVM`). |
| 31 | + - Backups are stored in the secondary storage. |
| 32 | + |
| 33 | +While by default the backups are stored in any secondary storage available, sharing the same space with templates, ISOs and snapshots, |
| 34 | +using the :ref:`Direct resources to a specific secondary storage` feature, it is possible to create backup-exclusive secondary storages. |
| 35 | + |
| 36 | +Currently, only backup of VMs from the NFS, File-based Shared Mountpoint and Local Storage based Primary Storage are supported. |
| 37 | + |
| 38 | +Since the KBOSS plugin is fully implemented by ACS, and does not depend on any outside provider, the **importBackupOffering** API is not used. |
| 39 | +Instead, use the **createBackupOffering**, which allows users to configure KBOSS features they want to use. These backup offerings are then |
| 40 | +assigned and used with KVM instances to perform support B&R actions and operations. |
| 41 | + |
| 42 | + |
| 43 | +Using the KBOSS Backup and Recovery Plugin |
| 44 | +------------------------------------------ |
| 45 | +To use the KBOSS Backup and Recovery Plugin, the Backup and Recovery framework needs to be enabled first. Then the backup plugin 'kboss' needs to be enabled |
| 46 | +on either the global or zone settings. |
| 47 | + |
| 48 | +================================= ======================== |
| 49 | +Configuration Value |
| 50 | +================================= ======================== |
| 51 | +backup.framework.enabled true |
| 52 | +backup.framework.provider.plugin kboss |
| 53 | +================================= ======================== |
| 54 | + |
| 55 | +Once the above two configurations are set, restart the cloudstack-management service. |
| 56 | +After, we can create the first backup offering. Navigate to the Service Offerings -> Backup Offerings tab. Click on 'Create Backup Offering' and fill the form. |
| 57 | + |
| 58 | +=============================== ======================== |
| 59 | +Field Value |
| 60 | +=============================== ======================== |
| 61 | +Name A suitable name to represent the Backup Offering. |
| 62 | +Description A suitable description to represent the Backup Offering. |
| 63 | +Zone Zone UUID for the backup offering. |
| 64 | +Allow User driven backups Whether to allow backups from the ACS API. If this value is false, you will not be able to create backups using KBOSS. **It must be set as true**. |
| 65 | +Public Whether the offering is public or dedicated to a domain. |
| 66 | +Domain Domain to dedicate the offering, only shown when 'Public' is false. |
| 67 | +Compress Whether the backups should be compressed after creation. |
| 68 | +Compression library Which compression library to use, only shown when 'Compress' is true. |
| 69 | +Validate Whether to validate backups after creation. |
| 70 | +Validation steps Which validation steps should be executed. |
| 71 | +Allow extract file Whether backups created with this offering should allow file extraction. This feature will be implemented in a future version. |
| 72 | +Allow quick restore Whether backups created with this offering should allow quick restore. |
| 73 | +Backup chain size The backup chain size of the incremental backups. This value overwrites what is set in `backup.chain.size`. |
| 74 | +=============================== ======================== |
| 75 | + |
| 76 | +.. image:: /_static/images/create-backup-offering.png |
| 77 | + :align: center |
| 78 | + :alt: NAS Backup repository |
| 79 | + |
| 80 | +After this has been done, you can go to any Instance view and assign the backup offering to it. Then, there will be buttons available for either ad-hoc |
| 81 | +backup or a scheduled backup of the VM. |
| 82 | + |
| 83 | +Quiesce (Filesystem Freeze and Thaw) |
| 84 | +------------------------------------ |
| 85 | + |
| 86 | +Users can set quiesce to true while creating a backup or a backup schedule. |
| 87 | +When a backup is initiated with quiesce enabled, CloudStack uses QEMU guest agent |
| 88 | +to freeze the filesystem before starting backup. This operation flushes all dirty |
| 89 | +filesystem buffers to disk and quiesces new writes. The filesystem is then thawed |
| 90 | +immediately after the backup process starts, keeping the freezing window very short. |
| 91 | + |
| 92 | +|NASB&R-quiesceInstance.png| |
| 93 | + |
| 94 | +This enhancement brings the KBOSS backup plugin from crash-consistent backups closer to |
| 95 | +application-consistent backups. |
| 96 | + |
| 97 | +Points to note: |
| 98 | + |
| 99 | +#. The feature requires Qemu Guest Agent to be installed and running on the guest instance. |
| 100 | +#. This method does not capture the memory state of the guest. Any data held in application memory |
| 101 | + that hasn’t been flushed to disk prior to the filesystem freeze will not be captured. |
| 102 | +#. For fully application-consistent backups, guest applications must implement pre-freeze hooks |
| 103 | + to flush their internal state to disk before the filesystem is frozen. |
| 104 | + |
| 105 | +Backup Chain Management |
| 106 | +----------------------- |
| 107 | + |
| 108 | +The size of the incremental backup chain is determined by the `backup.chain.size` zone-wide configuration; furthermore, it may be overwritten on the |
| 109 | +offering level. Once the backup chain reaches the configured size, the next backup will be a full backup and a new chain will start. |
| 110 | + |
| 111 | +If users wish to finalize the backup chain early, the **finishBackupChain** API may be used. It is also available through the VM instance interface on the GUI. |
| 112 | +If the VM is in the `BackupError` state, ACS will try to normalize it. Most times ACS is able to normalize it, however, the VM may be unable to be recovered |
| 113 | +automatically for a number of reasons, for example if the storage is unavailable. |
| 114 | + |
| 115 | +Isolated backups |
| 116 | +---------------- |
| 117 | + |
| 118 | +If you want to create a backup that does not depend on any backup chain, you may inform the `isolated` flag when creating a backup, or a backup schedule. |
| 119 | +When set, the created backup will be a full backup, with no dependencies on other backups, and no future backups will depend on it. |
| 120 | + |
| 121 | +Quick restore |
| 122 | +------------- |
| 123 | + |
| 124 | +If the backup was created using an offering with `allowQuickRestore` set, the user will be able to restore it using the `quickRestore` flag. |
| 125 | +When this is done, CloudStack creates new deltas on primary storage that have the backup as their backing files, while the backups are still on the |
| 126 | +secondary storage. Then, the VM is automatically started and the volume consolidation process starts. While this is done, the VM is already available to use. |
| 127 | + |
| 128 | +While this process is ongoing, the VM should not be stopped (from inside the VM), otherwise, it will fail to consolidate the volume. The benefit of using quick restore |
| 129 | +is that the VM is up and running in a very short time; however, the time to finish the consolidation process is bigger than normal restore. During |
| 130 | +the consolidation process, the VM's disk might have their performance slowed, when the process is done, the performance should go back to normal. |
| 131 | + |
| 132 | +Compression |
| 133 | +----------- |
| 134 | + |
| 135 | +The backup compression process is executed asynchronously after the backup is created, for VMs using backup offerings that support it. |
| 136 | +There are several configurations that affect the backup compression task and process: |
| 137 | + |
| 138 | +.. list-table:: Compression Configurations |
| 139 | + :widths: 20 60 20 |
| 140 | + :header-rows: 1 |
| 141 | + |
| 142 | + * - Configuration |
| 143 | + - Description |
| 144 | + - Default value |
| 145 | + * - `backup.compression.task.enabled` |
| 146 | + - Determines whether the task responsible for scheduling compression jobs is active. If not, compression jobs will not run |
| 147 | + - true |
| 148 | + * - `backup.compression.max.concurrent.operations.per.host` |
| 149 | + - Maximum number of concurrent compression jobs. Values lower than 1 disable the limit. |
| 150 | + - 5 |
| 151 | + * - `backup.compression.max.concurrent.operations` |
| 152 | + - Maximum number of compression jobs that can be executed at the same time in the zone. Values lower than 1 disable the limit. |
| 153 | + - 10 |
| 154 | + * - `backup.compression.max.job.retries` |
| 155 | + - Maximum number of attempts for executing compression jobs |
| 156 | + - 2 |
| 157 | + * - `backup.compression.retry.interval` |
| 158 | + - Interval, in minutes, between attempts to run compression jobs |
| 159 | + - 60 |
| 160 | + * - `backup.compression.timeout` |
| 161 | + - Timeout, in seconds, for running compression jobs |
| 162 | + - 28800 |
| 163 | + * - `backup.compression.minimum.free.storage` |
| 164 | + - Minimum required available storage to start the backup compression process. |
| 165 | + This setting accepts a real number that is multiplied by the total size of the backup to determine the necessary available space. |
| 166 | + By default, the storage must have the same amount of available space as the space occupied by the backup. This is checked by the host |
| 167 | + when trying to start the process. |
| 168 | + - 1 |
| 169 | + * - `backup.compression.coroutines` |
| 170 | + - Number of coroutines used for the compression process, each coroutine has its own thread |
| 171 | + - 1 |
| 172 | + * - `backup.compression.rate.limit` |
| 173 | + - Compression rate limit, in MB/s. Values less than 1 disable the limit |
| 174 | + - 0 |
| 175 | + |
| 176 | +To check information about compression jobs, the **listBackupServiceJobs** API may be added. |
| 177 | + |
| 178 | +Validation |
| 179 | +---------- |
| 180 | + |
| 181 | +The backup validation process is executed asynchronously after the backup is created, for VMs using backup offerings that support it. To perform the validation, a temporary dummy |
| 182 | +VM is created using the backup being validated and a few configurable validation steps are executed. After the first successful validation, a Hash of the backup is taken, |
| 183 | +a new Hash is taken periodically and compared with the original to make sure the backup is still valid. Furthermore, most backup validation steps are dependant on the guest VM having |
| 184 | +Qemu Guest Agent installed and configured to run on start. There are three currently available validation steps: |
| 185 | + |
| 186 | +.. list-table:: Validation steps |
| 187 | + :widths: 20 60 20 |
| 188 | + :header-rows: 1 |
| 189 | + |
| 190 | + * - Step |
| 191 | + - Description |
| 192 | + - Needs Qemu Gest Agent? |
| 193 | + * - `wait_for_boot` |
| 194 | + - Waits during a configurable timeout for the validation VM to boot. |
| 195 | + - Yes |
| 196 | + * - `execute_command` |
| 197 | + - Executes a configurable command inside the VM using Qemu Guest Agent, compares the result with a configurable expected result |
| 198 | + - Yes |
| 199 | + * - `screenshot` |
| 200 | + - Takes a screenshot of the VM's window after a configurable time. The screenshot can then be downloaded via `downloadValidationScreenshot` API. |
| 201 | + - No |
| 202 | + |
| 203 | +More validation steps may be added in the future. |
| 204 | + |
| 205 | +The configurations related to backup validation are detailed below: |
| 206 | + |
| 207 | +.. list-table:: Validation configurations |
| 208 | + :widths: 10 70 10 10 |
| 209 | + :header-rows: 1 |
| 210 | + |
| 211 | + * - Configuration |
| 212 | + - Description |
| 213 | + - Default value |
| 214 | + - Scope |
| 215 | + * - `backup.validation.task.enabled` |
| 216 | + - Determines whether the task responsible for scheduling validation jobs is active. If it is not active, validation jobs will not run. |
| 217 | + - `true` |
| 218 | + - Account |
| 219 | + * - `backup.validation.interval` |
| 220 | + - Interval, in hours, between two validations of the same backup. |
| 221 | + - `24` |
| 222 | + - Account |
| 223 | + * - `backup.validation.max.concurrent.operations` |
| 224 | + - Maximum number of validation jobs that can be executed at the same time in the zone. Values lower than 1 disable the limit. |
| 225 | + - `10` |
| 226 | + - Zone |
| 227 | + * - `backup.validation.max.concurrent.operations.per.host` |
| 228 | + - Maximum number of validation jobs that can be executed at the same time on each host. Values lower than 1 disable the limit. |
| 229 | + - `1` |
| 230 | + - Cluster |
| 231 | + * - `backup.validation.boot.default.timeout` |
| 232 | + - Default timeout, in seconds, for the boot validation step |
| 233 | + - `240` |
| 234 | + - Account |
| 235 | + * - `backup.validation.script.default.timeout` |
| 236 | + - Default timeout, in seconds, for the script validation step |
| 237 | + - `60` |
| 238 | + - Account |
| 239 | + * - `backup.validation.screenshot.default.wait` |
| 240 | + - Default waiting time, in seconds, before executing the VM screenshot |
| 241 | + - `60` |
| 242 | + - Account |
| 243 | + * - `backup.validation.end.chain.on.fail` |
| 244 | + - If true, ends the current backup chain if backup validation fails and the backup belongs to that chain. |
| 245 | + - `true` |
| 246 | + - Account |
| 247 | + * - `enforce.resource.limit.on.backup.validation.vm` |
| 248 | + - If true, the creation of validation VMs is bound by the account/domain resource limits. |
| 249 | + - `false` |
| 250 | + - Account |
| 251 | + |
| 252 | +To configure the validation steps on a VM level, new VM settings were created. Most of them are available to end-users by default, except the timeouts. The availability of |
| 253 | +these settings to User accounts may be configured by the `user.vm.readonly.details` global configuration. It is expected that users use these settings to configure the validation |
| 254 | +process, especially the `execute_command` step, which if not configured using the VM settings will not run. Below is a table of the VM settings pertaining to the backup |
| 255 | +validation process: |
| 256 | + |
| 257 | +.. list-table:: Validation VM settings |
| 258 | + :widths: 20 80 |
| 259 | + :header-rows: 1 |
| 260 | + |
| 261 | + * - Setting |
| 262 | + - Description |
| 263 | + * - `backupValidationCommand` |
| 264 | + - Command to be executed during the command execution step. The step will not run if this configuration does not have a value |
| 265 | + * - `backupValidationCommandArguments` |
| 266 | + - Arguments to be passed to the command executed during the command execution step |
| 267 | + * - `backupValidationCommandExpectedResult` |
| 268 | + - Expected result of the command execution, coded in Base64. If not provided, the process exit code will be checked for value `0` |
| 269 | + * - `backupValidationCommandTimeout` |
| 270 | + - Timeout for the command executed during the command execution step. Overrides the global configuration |
| 271 | + * - `backupValidationScreenshotWait` |
| 272 | + - Waiting time before executing the screenshot during the screenshot step. Overrides the global configuration |
| 273 | + * - `backupValidationBootTimeout` |
| 274 | + - Timeout for the VM boot. Overrides the global configuration |
| 275 | + |
| 276 | +To check information about validation jobs, the **listBackupServiceJobs** API may be added. |
| 277 | + |
| 278 | + |
| 279 | +.. |NASB&R-quiesceInstance.png| image:: /_static/images/NASB&R-quiesceInstance.png |
| 280 | + :alt: Quiesce option while creating backups. |
| 281 | + :width: 400 px |
0 commit comments