“The maximum limit for OpenStorage devices and their concurrent write operations has been exceeded.” Deduplication folder goes offline after multiple Deduplication folders are shared with Backup Exec servers.

Problem

The Deduplication folder goes offline after multiple Deduplication folders are shared between Backup Exec MBES and/or CAS servers.

Or

Sharing or Creation of Deduplication\MSDPC\Cloud storage fails with following error.

The maximum limit for OpenStorage devices and their concurrent write operations has been exceeded. Reduce the number of OpenStorage devices and/or their concurrent write operations. To reduce the number of concurrent write operations, edit the properties of an OpenStorage device.

Error Message

Or

The local Deduplication folder appears offline in the storage tab and jobs targeted to deduplication storage folder fail with the following error:  

Robotic library hardware error. 

The robotic library has reported a general hardware error condition. Manual intervention, calibration, etc., may be required. This may be a device related hardware problem, or it may be some other hardware problem (i.e. SCSI card/bus etc.). The library and drive states have been set to offline. Please attend to this condition. 

Errors observed in ADAMM log:

PvlChanger::MapErrorCode() - offline device  

Drive = 1246 "Dedupe 0001:57" 

ERROR = 0xE000820A (E_CHG_HARDWARE) 

PvlDrive::UpdateOnlineState() 

Drive = 1247 "Dedupe 0001:58" 

ERROR = The device is offline! 

Cause

There have been some changes made in the internal concurrency of deduplication devices in recent releases to provide better performance for parallel Microsoft 365 backup jobs. This resulted in changing the number of supported shared deduplication devices per server. This will affect Deduplication folders, OST and MSDPC devices.

For fresh installation of BE 25.1 ->

A maximum of 1 local and 2 shared deduplication devices can remain online when Deduplication concurrent device per server limit is exceeded. There will still be some concurrency available for some more OST and MSDPC devices to be added.

For upgrade from BE 23, BE 24, BE 25 to BE 25.1 ->

A maximum of 1 local and 4 shared deduplication devices can remain online when Deduplication concurrent device per server limit is exceeded. No more OST and MSDPC devices can be added.

Solution

Un-share one of the Deduplication folders and restart BE services. Once the concurrent devices are within the limit, sharing or adding new MSDPC storage will work.

Or

Set the “Concurrency operations:” between 2 to 8. As shown in below image and restart BE service.

Or

Create following registry key on affected BE servers (CAS and MBES server) and restart BE services –

HKEY_LOCAL_MACHINE\SOFTWARE\Software\Veritas\Backup Exec For Windows\Adamm

Type= REG_DWORD

ValueName = "ExtendConcurrencyTo"

Value = 32.

Note: “ExtendConcurrencyTo” registry value applies to Deduplication storage. The value of 32 indicates that each Deduplication device will be created with 32 concurrency. Microsoft 365 Forever incremental and Consolidate Full jobs have the requirement of higher concurrency to support multi-streaming.

In case you have Microsoft 365 job configured and plan to run parallel jobs, it is recommended to have 1 Local and 2 shared deduplication device configured on a given managed server in the CAS-MMS setup. Having more than the recommended number of deduplication devices will impact Microsoft 365 job performance.

Applies to -->  Backup Exec 24, Backup Exec 25