NetBackup™ Web UI Cloud Administrator's Guide

Troubleshoot PaaS workload protection and recovery issues

If you manually delete a backup image created by an incremental schedule, the next incremental backup may either fail or run as a full backup. The outcome depends on which image was removed.

For example, consider a backup sequence with images F1 (full), I1, I2, and I3 (incremental).

The error message specifies the expected number of images (X) and the actual number found (Y) in the catalog.

You can see the following message in the Activity Monitor:

AuthorizationFailed -Message: The client '<clientId> '<objetId>' does not have authorization to perform action 'Microsoft.Sql/servers/databases/read' over scope '<resoourceId>' or the scope is invalid. If access was recently granted, refresh your credentials.

Explanation: This error occurs when the Snapshot Manager and NetBackup are deployed in AKS, and:

Workaround: Do any of the following:

Explanation: This issue occurs if the Read-only lock or Delete lock attribute is applied to the database or the resource group.

Workaround: Before performing any backup or restore, remove any existing Read-only lock and Delete lock attributes from the database or the resource group.

Explanation: This appears when you manually cancel a backup or a restore job from the activity monitor and a database is created on the portal during the partial restore operation.

Workaround: Manually clean up the database on the provider portal, and temporary staging location at the universal share mount location under a specific directory created by the database name.

Explanation: If the Snapshot Manager container service restarts abruptly; the provider-protected restore jobs may remain in the active state and you may not see the updated status on the activity monitor details page.

Workaround: Restart the workflow containers using the following command in the Snapshot Manager:

docker restart flexsnap-workflow-system-0-min flexsnap-workflow-general-0-min

After restarting the containers, the restore jobs are updated with the latest status in the activity monitor.

Explanation: Appears if the client name used for the backup exceeds the length of 255 characters.

The bpdbm log confirms the same by displaying the following error message:

db_error_add_to_file: Length of client is too long. Got 278, but limit is 255. read_next_image: db_IMAGEreceive() failed: text exceeded allowed length (225)

Workaround: Rename the database such that the client name fits within the length of 255 characters.

Explanation: This occurs when the media server is running a lower version of NetBackup or is hosted on an unsupported platform.

Workaround: You must have at least one media server with the following configuration:

You can either add a new media server with the supported version and platform, or upgrade an existing one to meet these requirements.

Explanation: Job details include the following error:

An unexpected failure occurred: Data plan execution failed with the message - "One or more errors occurred. (One or more errors occurred. (One or more errors occurred. (Exception of type 'System.OutOfMemoryException' was thrown.")

Workaround: This issue is caused by a limitation in Microsoft's SqlPackage utility.

To resolve it, increase the available memory on the media server to match the database size, and then retry the job.

There is no exact formula to calculate the required memory, but it is recommended to allocate at least twice the memory of the database schema size.

Or,

Explanation:Occurs during backup if the policy prefix length during protection plan creation is larger than the allowed length. Due to this the file path length of the catalog image exceeds 256 chars and fails with the above error message in the Activity monitor.

The bpdbm log confirms the same by displaying the following error message:

<16> db_error_add_to_file: cannot stat(\\?\C:\Program Files\Cohesity NetBackup\NetBackup\db\images \azure-midb-1afb87487dc04ddc8fafe453dccb7ca3+ nbux-qa-bidi-rg+eastus+az-sql-mi-bidinet01+ testdb_bidinet02\1656000000\tmp\catstore\ BACKUPNOW+141a73e7-cdc4-4371-823a-f170447dba2d_ 1656349831_FULL.f_imgUserGroupNames0): No such file or directory (2) <16> ImageReadFilesFile::get_file_size: cannot stat(\\?\C:\Program Files\Veritas\NetBackup\db \images\azure-midb-1afb87487dc04ddc8fafe453d ccb7ca3+nbux-qa-bidi-rg+eastus+az-sql-mi-bidinet01+testdb_ bidinet02\1656000000\tmp\catstore\BACKUPNOW+141a73e7-cdc4-4371 -823a-f170447dba2d_1656349831_FULL.f_imgUserGroupNames0): No such file or directory (2) <16> ImageReadFilesFile::executeQuery: Cannot copy \\?\C:\Program Files\Veritas\NetBackup\db\images\azure-midb-1afb87487dc04ddc8fafe453dccb7 ca3+nbux-qa-bidi-rg+eastus+az-sql-mi-bidinet01+testdb_bidinet02\1 656000000\tmp\catstore\BACKUPNOW+141a73e7-cdc4-4371-823a-f170447d ba2d_1656349831_FULL.f_imgUserGroupNames0

Workaround: Use a policy prefix name in the protection plan with length less than 10 characters, so that the total length of the catalog path is less than 256 characters.

Explanation:NetBackup is not able to successfully carry out the requested operation.

Recommended action: Refer to the activity monitor details for the possible reasons for failure.

Explanation: The error message seen in dbagentsutil logs as,pg_dump: error: query failed: ERROR: permission denied for table test;pg_dump: error: query was: LOCK TABLE public.test IN ACCESS SHARE MODE;Invoked operation: PRE_BACKUP failed

Occurs when you try to back up a database that has multiple tables with different roles. If tables have at least one different owner, other than the database owner, and it is not a member of the database owner role, then the backup may fail.

Recommended action: You must have a role that has access to all tables inside the database that you want to backup or restore.

For example, say that we wanted to back up the School database which has two tables.

Create a new role. Say, NBUbackupadmin

Run the following command to create the role:

postgres=> CREATE USER NBUbackupadmin WITH PASSWORD '***********';

CREATE ROLE

To make this new role a member of postgres and schooladmin role, run:

postgres=> GRANT postgres TO NBUbackupadmin;

GRANT ROLE

postgres=> GRANT schooladmin TO NBUbackupadmin;

GRANT ROLE

Explanation: Backups fail due to loss of connectivity to the media server.

Recommended action: You can restart the backup job if the policy has checkpoints enabled. Once the network issue is resolved, select the incomplete backup job in the web UI and click Resume. The job resumes from the point it was stopped. If the checkpoint is not enabled in the policy, the job shows up as a failed job in the web UI.

Explanation: The Job details contain additional details: ManagedIdentityCredential authentication unavailable. The requested identity is not assigned to this resource. The allocated media server does not have any Managed Identity attached to it.

Recommended action: When using system or user-managed identity for the PaaS Azure SQL and Managed Instances, apply the same permissions and rules to the media server(s) and the Snapshot Manager. If you use user-managed identity, attach the same user-managed identity to the media server(s) and the Snapshot Manager.

Differential incremental backup is supported for only for Azure SQL server and Azure SQL Managed Instance. When you select an unsupported backup type, this error appears.

Explanation: Backup failed due to an error while executing the backup query script on the Azure SQL Server. The following error message appears:

('42S02', "[42S02] [Microsoft][ODBC Driver 18 for SQL Server] [SQL Server]Invalid object name 'cdc.<schema_name>_<table_name>_CT'. (208) (SQLExecDirectW)")

This issue may occur due to a mismatch between user tables and their associated CDC tables. The mismatch typically arises after schema changes or when database objects are moved between schemas after CDC is enabled.

Recommended action:

NetBackup automatically re-enables CDC during the next backup after you manually disable it.

Appears when you do not have permissions to enable or disable the CDC.

Workaround: Give NetBackup the necessary permissions to enable or disable the CDC in your Azure environment.

Explanation: Occurs during restore if the backup image is generated on 10.2 media and restore goes to an older (< 10.2) media server.

Workaround: Change the restore media to 10.2 and remove the older media from storage.

Explanation: Currently the AWS API response does not show if a table has auto scaling enabled. So, during backup, this metadata is not captured in NetBackup, and as a result the restored table does not have auto scaling enabled.

Workaround: Enable the auto-scaling property of the restored DynamoDB table in the AWS portal manually.

Explanation: Azure SQL MI maintains CDC-enabled database details in the table cdc_jobs inside the msdb schema. When the database is dropped, its cdc_jobs entry should be deleted. Sometimes this entry does not get deleted from the cdc_jobs table. So, when a new database is created with the same db_id which already exists in the cdc_jobs table, the issue occurs.

Workaround: When you drop a database, check the entry of the dropped database in the cdc_jobs table of the msdb schema. If the entry is present there, delete it manually.

Explanation: Failure of the RDS boto3 APIs shows this error. NetBackup shows this error for the DescribeDBInstances operation.

Workaround: Synchronize the date and time of the media server with the actual network date and time.

Also, verify if you are using the correct provider credentials.

Explanation: The import operation on the target domain may fail with the state code 191: No images successfully processed. The job details in the Activity monitor show: Failed to create the JSON payload.

Cause: The image that you are replicating to the target domain is created from a NetBackup 10.4 or older media server, which does not have the required metadata in the NetBackup catalog.

Workaround: Do any of the following: