Common issues / Configuration recommendations for Veritas NetBackup for SAP HANA

Description

Common issues / Configuration recommendations for Veritas NetBackup for SAP HANA

Symptom

The following are recommendations for Veritas NetBackup for SAP HANA, or  when encountering issues when trying to backup or restore SAP HANA using the backint interface with Veritas NetBackup.

• Backup, Recovery, NetBackup, Veritas, Backint, SAP HANA
• Reason and Prerequisites: A SAP-certified solution of Veritas NetBackup and a SAP HANA database.

Solution

The following documents give recommendations on how to configure Veritas NetBackup for SAP HANA, or describe common issues with Veritas NetBackup and how to resolve them.

Configuration Recommendations:

    Increase the Client Read Timeout in NetBackup to a large value. If the Client Read Timeout is set too low, the SAP HANA restore job may fail with error code 2850 and the Job Details tab shows status 41 (network connection timeout error).

    --> Increase the value for Client Read Timeout of NetBackup media server and NetBackup client servers to 3600 seconds (approximately 1 hours) or more.

    From the NetBackup Administration console:

•     a. (NetBackup media server) Select Host Properties > Media Servers, and then double-click the media server to open Media Server Properties dialog box.
•     b. (NetBackup client server) Select Host Properties > Clients Servers, and then double-click the client server to open Clients Server Properties dialog box.

    a. Select the Timeouts tab.

    b. Modify the default value of Client Read Timeout to 3600 seconds (approximately 8 hours) or more.

    c. Click OK.

    If restoring from tape, set the parameter MAX_RECOVERY_BACKINT_CHANNELS from the default value of '64' to a value of '1' in SAP HANA Studio. If this parameter is not set accordingly, sometimes a log restore from tape may hang until cancelled or eventually timeout and fail.

•     See: https://www.veritas.com/support/en_US/article.TECH211556
•     Scheduling SAP HANA backups from NetBackup : https://www.veritas.com/support/en_US/article.TECH209343
•     Scheduling SAP HANA delta backups (differential and incremental) from NetBackup: https://www.veritas.com/support/en_US/article.000025716
•     SAP HANA Multi-Tenant Database Containers Backups and Recovery using NetBackup: https://www.veritas.com/support/en_US/article.000109387

Common configuration issues:

•     If the library libelf.so.0 file is missing on the SAP HANA appliance (known for IBM and HITACHI SAP HANA appliances), SAP HANA backup fails with exit code 127. See SAP Note 1978901 or Veritas article: https://www.veritas.com/support/en_US/article.TECH208377
•     SAP HANA backup fails almost immediately if the initSAP.utl file has not been configured correctly ("ERROR: switch_list missing from parameter file") See: https://www.veritas.com/support/en_US/article.100012392

More technical information is available through the Veritas Knowledge Base Search available at https://www.veritas.com/support/ (search for "SAP HANA").

Troubleshooting Backups with Third-Party Backup Tools

1. Troubleshooting procedures for common issues involving Backint.
Out of Memory Error when Using Backint: When using Backint, this error is seen in the esserver trace files:

Executed: hdbnsutil -setServiceStopping --serviceType=esserver --port=30212 --pid=36153 without success (popen failed, errno=12)

Errno=12 indicates an out of memory condition.

Reason:

The system does not have enough swap space for Backint.

Solution:

Increase the servers swap space. The swap space must be at least two times the size of the extended storage caches.

    a. Open the esserver.ini file.
    b. Sum these cache sizes:

        main_cache_mb

        temporary_cache_mb

        delta_memory_mb

        heap_memory_mb
    c. Double this result and ensure the swap space is at least as large as the result in step b.

2. Slow Backint Processes

Backint appears to hang. This error may appear in esserver.out:

[00000]{0000000000}[-1/-1] 2017-11-06 20:10:03.518465 i esserver UNKNOWN(0) : Opening of Pipe /usr/sap/ABC/SYS/global/hdb/backint/DB_MWD/log_backup_1024_20171106_181000035, failed with errno 6, retrying

Reason:
If hdbbackint does not open the pipe within a one minute window, but connects shortly afterward, there's a possibility the hdbbackint process will continue listening after dynamic tiering has timed out. In such a situation, the hdbbackint process appears to hang.
Note This issue was fixed in recent revisions of SAP HANA dynamic tiering 1.0 (see 2375865 Information published on SAP site) and SAP HANA dynamic tiering 2.0. (see 2447889 Information published on SAP site).

Solution:

• Upgrade to a recent revision.
• If upgrading is not practical, then terminate the hdbbackint process and trigger another backup.

3. Error in Backint.log: No Such File or Directory

When using Backint,  this error is seen in backint.log:

Backint cannot execute /usr/sap/ABC/SYS/global/hdb/opt/hdbbackint, No such file or directory (2)

Reason:

Backint is not accessible to the dynamic tiering host.

Solution:

• Check permissions. Backint may not have executable permissions. Ensure Backint has executable permissions on the dynamic tiering host.
• Check the installation status. Backint may not have been installed properly. Reinstall if necessary.

4. Error in Backup.log: Unable to Execute Command on Name Server

When using Backint,  this error is seen in backup.log:

[447]backup could not be completed, [110112] Extended storage backup error : [SAP][ODBC Driver]Error during backup/restore: HANA environment error, unable to execute command on name server

Reason:

The name server is not running. A running name server is required on each SAP HANA host.

The name server process may have been terminated, or the name server might have gone offline.

Solution:

Restart the name server.

For more information, see the following:

 https://help.sap.com/viewer/269740c67eca42a3b4ffbd376b406fbe/2.0.03/en-US/bd5312aade814b04835e9991e6e86402.html