NDMP restores fail with media error EXIT STATUS 85

Problem

Performing NDMP restores using a Hitachi NDMP filer from DataDomain OST fail with status 85 (regardless of date of image selected) unless the image is duplicated off to tape then a restore will complete with success.
 

Error Message

Example Detailed Status from the Activity Monitor:

9/20/2013 11:36:10 AM - Critical bptm(pid=4712) sts_read_image failed: error 2060046 bytes Read = 0plugin error
9/20/2013 11:36:10 AM - Critical bptm(pid=4712) image read failed: error 2060046: plugin error
9/20/2013 11:36:10 AM - Error bptm(pid=4712) cannot read image from disk, Invalid argument
9/20/2013 11:36:10 AM - Error ndmpagent(pid=11440) NDMP restore failed from path /__VOLUME__/VOL
9/20/2013 11:36:10 AM - Info bptm(pid=4712) EXITING with status 85 <----------
9/20/2013 11:36:10 AM - Info ndmpagent(pid=11440) done. status: 150
9/20/2013 11:36:11 AM - restored image 192.168.0.2_1379620817 - (media read error(85)); restore time 00:00:29
9/20/2013 11:36:15 AM - end Restore; elapsed time: 00:00:34
media read error(85)

Example bptm log snippet from the media server:

11:35:47.282 [4712.6328] <2> read_config_file: using 25 value from D:\Program Files\Veritas\NetBackup\db\config\NUMBER_DATA_BUFFERS
11:35:47.282 [4712.6328] <2> read_config_file: using 32 value from D:\Program Files\Veritas\NetBackup\db\config\NUMBER_DATA_BUFFERS_DISK
11:35:47.282 [4712.6328] <2> read_backup: using 32 data buffers
11:36:09.872 [4712.6328] <2> read_data: read short block, 312 bytes, sending to socket
11:36:10.059 [4712.6328] <2> bp_sts_read_image_non_512: adjusting offset or length to a 512 byte boundry
11:36:10.059 [4712.6328] <16> 223983:bptm:4712:clientname.org: D:\Program Files\Veritas\NetBackup\bin\\ost-plugins\libstspiDataDomain.dll:stspi_rea
d_image STS_EPLUGIN [DDErrNo = 5008 (invalid argument)]
11:36:10.137 [4712.6328] <32> bp_sts_read_image_non_512: sts_read_image failed: error 2060046 bytes Read = 0
11:36:10.215 [4712.6328] <32> read_data: image read failed: error 2060046:
11:36:10.324 [4712.6328] <16> read_data: cannot read image from disk, Invalid argument
11:36:10.777 [4712.6328] <2> bptm: EXITING with status 85 <----------

Example Data Domain log (ddfs.info) snippet:

09/20 11:22:53.364 (tid 0x2aaaab112130): ddboost-<192.168.0.21-64800>: JOB START IMAGE_READ ip=192.168.0.210 pid=1220 cd=0
09/20 11:22:53.365 (tid 0x2aaaab112130): DDBOOST 192.168.0.210 (veth1) Jobs IMAGE_WRITE=1, IMAGE_READ=1, OPTDUP_SOURCE=0 OPTDUP_TARGET=0,
09/20 11:22:53.365 (tid 0x2aaaab112130): DDBOOST 192.168.0.210 (veth1) Jobs IMAGE_WRITE=1, IMAGE_READ=1
09/20 11:22:53.367 (tid 0x2aaaab112130): sfs_lookup(“/data/DD890/192.168.0.2_1379620817_C1_F1:1379620817:NAS:4:0::”, 0x88000000)
09/20 11:23:09.758 (tid 0x2aaaab112130): WARNING: ddboost-<192.168.0.21-64800>: ddboost_api ERROR: ddp_read() failed Offset 3651072, BytesToRead 1049088, BytesRead 0 Err: 5008-Invalid Input
09/20 11:23:09.758 (tid 0x2aaaab112130): WARNING: ddboost-<192.168.0.21-64800>: plugin ERROR: D:\Program Files\Veritas\NetBackup\bin\\ost-plugins\libstspiDataDomain.dll:stspi_read_image STS_EPLUGIN [DDErrNo = 5008 (invalid argument)]

Cause

The Hitachi NDMP filer requests to position directly where the image is stored on the OST disk.  OST only works with 512k byte boundaries. For example on Hitachi, if the file needed for restore is at position 612 of the backup, the filer would request the OST device to move move directly to 612, this is an issue with OST due to being outside of the 512k boundary.

With the introduction of NDMP multiplexing code was introduced to NetBackup to work with the above mentioned Architectural difference which affected customers using Hitachi filers and a SIZE_DATA_BUFFER_DISK at 1048576 = 1mb.
 

Solution

Workaround #1:

• Reduce the SIZE_DATA_BUFFERS_DISK to a size less than 1048576.
  • Note: NDMP works on 1024 block sizes. Recommend 524288 (512k).
• This touchfile can be adjusted at the following locations:
  • Windows: install_path\Veritas\NetBackup\db\config\SIZE_DATA_BUFFERS_DISK
  • UNIX/Linux: /usr/openv/netbackup/db/config/SIZE_DATA_BUFFERS_DISK

Workaround #2:

• Duplicate the image to NDMP tape and restore.

Workaround #3

• If workaround 1 and/or 2 do not work try disabling DAR
• In the NetBackup Administration Console, expand Host Properties and click Master Servers or Media Servers.
• Right-click the name of the server and select Properties.
• Click General Server.
• Uncheck the Use direct access recovery for NDMP restores box and click Apply.

  This action disables DAR on all NDMP restores.

You can re-enable DAR after the restore has completed,

Veritas has acknowledged that the above mentioned issue (Etrack  3371173) is present in the current version(s) of the product(s) mentioned at the end of this article. Veritas is committed to product quality and satisfied customers. Currently there is no ETA of a fix other than the mentioned workarounds.

Please note that Veritas Corporation reserves the right to remove any fix from the targeted release if it does not pass quality assurance tests or introduces new risks to overall code stability. Veritas's plans are subject to change and any action taken by you based on the above information or your reliance upon the above information is made at your own risk.

Applies To

• NetBackup 7.5 - 7.6
• Hitachi NDMP Filer
• DataDomain OST storage device
• SIZE_DATA_BUFFERS_DISK or SIZE_DATA_BUFFERS = 1048576 (1mb)