Backup with Variable-Length Deduplication enabled fails if SIZE_DATA_BUFFERS is equal to or greater than 2MB
Problem
Backup with Variable-Length Deduplication and multi-threaded agent enabled fails with status 84 if SIZE_DATA_BUFFERS is equal to or greater than 2MB
Error Message
Job details will show the following sequence, confirming that the value in SIZE_DATA_BUFFERS is at or above 2MB, multi-threaded agent is used and VLD is enabled::
Aug 17, 2018 5:55:04 AM - Info bptm (pid=7176) start
Aug 17, 2018 5:55:04 AM - Info bptm (pid=7176) using 2097152 data buffer size
Aug 17, 2018 5:55:04 AM - Info bptm (pid=7176) using 30 data buffers
Aug 17, 2018 5:55:04 AM - Info bptm (pid=7176) start backup
Aug 17, 2018 5:55:05 AM - begin writing
Aug 17, 2018 5:55:06 AM - Critical bptm (pid=7176) Storage Server Error: (Storage server: PureDisk:rhelref) mtstrm_write_segment: Fatal error occured in
Multi-Threaded Agent: Could not fit header (type=1) with data length 2097152 into the batch, even after batch has been flushed V-454-95
Aug 17, 2018 5:55:06 AM - Critical bptm (pid=7176) image write failed: error 2060017: system call failed
Aug 17, 2018 5:55:06 AM - Info bpbkar (pid=7165) bpbkar waited 0 times for empty buffer, delayed 0 times
Aug 17, 2018 5:55:09 AM - Error bptm (pid=7176) cannot write image to disk, Invalid argument
Aug 17, 2018 5:55:09 AM - Info bptm (pid=7176) EXITING with status 84 <----------
Aug 17, 2018 5:55:12 AM - Info rhelref (pid=7176) StorageServer=PureDisk:rhelref; Report=PDDO Stats (multi-threaded stream used, VLD enabled) for (rhelref):
scanned: 8221 KB, CR sent: 190 KB, CR sent over FC: 0 KB, dedup: 97.7%, cache hits: 838 (98.9%)
Aug 17, 2018 5:55:13 AM - Info bpbkar (pid=7165) done. status: 84: media write error
Aug 17, 2018 5:55:13 AM - end writing; write time: 0:00:08
media write error (84)
Solution
The default value of the BackupShmBufferSize setting in /usr/openv/lib/ost-plugins/mtstrm.conf file is 2MB which is not sufficient for SIZE_DATA_BUFFERS setting of 2MB or greater. To resolve the issue, set this value to (SIZE_DATA_BUFFERS / 1024) + 1 and run this command for the change to take effect on the next backup:
/usr/openv/pdde/pdag/bin/mtstrmd -terminate
For example, if the value in SIZE_DATA_BUFFERS is4194304 then set BackupShmBufferSize to 5, because (4194304 / 1024) + 1 = 5 (MB).
Alternate workarounds - make any of the following changes:
-
Reduce the value in SIZE_DATA_BUFFERS and/or SIZE_DATA_BUFFERS_DISK file to 1048576 or lower, or
-
Disable the usage of multi-threaded agent by setting MTSTRM_BACKUP_ENABLED to 0 in pd.conf, or
-
Disable Variable-Length Deduplication
There are currently no plans to address this issue (ET3956532) by way of a patch or hotfix in the current or previous versions of the software at the present time. While this issue may be resolved in a future major revision of the software at a later time, it is not currently scheduled for any release. If you feel this issue has a direct business impact for you and your continued use of the product, please contact your Veritas Sales representative or the Veritas Sales group to discuss these concerns.