Duplications to MSDP-Cloud using Azure Archive Tier fails with "One of the HTTP headers specified in the request is not supported"

Problem

When duplicating (or backing up) to an MSDP-Cloud LSU which utilizes the Azure Archive Tier for storage it can fail with the One of the HTTP headers specified in the request is not supported error if the region does not support the archive tier - for example "West US 3" region (that is due to be supported in Q1 2023 at time of writing).

Error Message

Job Description from activity monitor.

Oct 27, 2022 6:57:08 PM - Critical bpdm (pid=3830679) Storage Server Error: (Storage server: PureDisk:nbmedia01) async_get_job_status: Replication started but failed to complete successfully: Error occurred during replication. Look at the replication logs on the source storage server for more information. V-454-105
Oct 27, 2022 6:57:08 PM - Critical bpdm (pid=3830679) sts_copy_extent failed: error 2060014 operation aborted
Oct 27, 2022 6:57:09 PM - Error bpduplicate (pid=1938925) host nbmedia01 backup id client01_1666875585 optimized duplication failed, termination requested by administrator (150).
Oct 27, 2022 6:57:09 PM - Error bpduplicate (pid=1938925) Duplicate of backupid client01_1666875585 failed, termination requested by administrator (150).
Oct 27, 2022 6:57:09 PM - Info nbmedia01 (pid=3830679) StorageServer=PureDisk:nbmedia01; Report=PDDO Stats for (nbmedia01): scanned: 0 KB, CR sent: 0 KB, CR sent over FC: 0 KB, dedup: 0.0%, cache disabled, where dedup space saving:0.0%, compression space saving:0.0%
Oct 27, 2022 6:57:09 PM - Error bpduplicate (pid=1938925) Status = no images were successfully processed.
Oct 27, 2022 6:57:09 PM - end Duplicate; elapsed time 0:03:03
termination requested by administrator  (150)

 BPTM log:
 
 11:45:34.777 [2590980] <2> ConnectionCache::connectAndCache: Acquiring new connection for host primary01, query type 1 (Q_ERRADD)
 11:45:34.813 [2590980] <4> db_sendrequest: Sending request(cached=true) for query_type:[1:Q_ERRADD] on socket 0(BPDBM CONNECT FROM x.x.x.x.42047 TO x.x.x.x.1556 fd = 0)
 11:45:34.824 [2590980] <2> closeConnection: Caching connection (BPDBM CONNECT FROM x.x.x.x.42047 TO x.x.x.x.1556 fd = 0) for query type [1:Q_ERRADD] for reuse, request stats(reuseCount=0, requestTime=0.011447)
 11:45:44.150 [2590980] <16> 1082:bptm:2590980:nbmedia01: [ERROR] PDSTS: impl_set_imh_image_prop: PdvfsClose() failed (5:Input/output error)
 11:45:44.179 [2590980] <16> 1082:bptm:2590980:nbmedia01: [ERROR] PDSTS: pi_close_image_v7: impl_close_image() failed (2060017:system call failed)
 11:45:44.179 [2590980] <16> 1082:bptm:2590980:nbmedia01: [ERROR] PDSTS: stspi_close_image_v7: stspi_close_image_v7 failed with error <2060017>, client:, policy:, image:
 11:45:44.180 [2590980] <2> ConnectionCache::connectAndCache: Acquiring new connection for host primary01, query type 1 (Q_ERRADD)
11:45:44.209 [2590980] <4> db_sendrequest: Sending request(cached=true) for query_type:[1:Q_ERRADD] on socket 0(BPDBM CONNECT FROM x.x.x.x.40549 TO x.x.x.x.1556 fd = 0)

ocsd log:

{"level":"error","File":"MEDIA01_MSDP/MSDP_DATA/data/ds_5/upload/360287970189648880.bin","Key":"nbmedia01-BLOB-ARCH_VL/data/328/351843720888328/360287970189648880.sobin","Status code":213,"errmsg":"-> github.com/Azure/azure-storage-blob-go/azblob.newStorageError, /extsrc/go_sdk/azure/0.14d/src/github.com/Azure/azure-storage-blob-go/azblob/zc_storage_error.go:42\n===== RESPONSE ERROR (ServiceCode=UnsupportedHeader) =====\nDescription=One of the HTTP headers specified in the request is not supported.\nRequestId:ac9ba2aa-e01e-0028-5a9a-ee5cd4000000\nTime:2022-11-02T09:08:05.8295174Z, Details: \n   Code: UnsupportedHeader\n   HeaderName: x-ms-access-tier\n   PUT https://nbuarchbkp.blob.core.windows.net:443/media01/nbmedia01-BLOB-ARCH_VL/data/328/351843720888328/360287970189648880.sobin/0?comp=blocklist\n   Authorization: REDACTED\n   Content-Length: [64]\n   Content-Type: [application/xml]\n   User-Agent: [APN/1.0 Veritas/1.0 NetBackup/MSDPCloud 16.0001.0022.0527]\n   X-Ms-Access-Tier: [Archive]\n   X-Ms-Blob-Cache-Control: []\n   X-Ms-Blob-Content-Disposition: []\n   X-Ms-Blob-Content-Encoding: []\n   X-Ms-Blob-Content-Language: []\n   X-Ms-Blob-Content-Type: [text/plain]\n   X-Ms-Client-Request-Id: [93396ac7-3f84-4b85-75a0-e58487498f64]\n   X-Ms-Date: [Wed, 02 Nov 2022 09:08:05 GMT]\n   X-Ms-Meta-Archive: [true]\n   X-Ms-Version: [2020-08-04]\n   --------------------------------------------------------------------------------\n   RESPONSE Status: 400 One of the HTTP headers specified in the request is not supported.\n   Content-Length: [293]\n   Content-Type: [application/xml]\n   Date: [Wed, 02 Nov 2022 09:08:04 GMT]\n   Server: [Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0]\n   X-Ms-Client-Request-Id: [93396ac7-3f84-4b85-75a0-e58487498f64]\n   X-Ms-Error-Code: [UnsupportedHeader]\n   X-Ms-Request-Id: [ac9ba2aa-e01e-0028-5a9a-ee5cd4000000]\n   X-Ms-Version: [2020-08-04]\n\n\n","Elapsed time":"11.309287274s","time":"2022-11-02T14:38:05.828308191+05:30","message":"Upload failed"}
{"level":"error","handleRequest":"Upload failed, could not try more times","Tried times":11,"time":"2022-11-02T14:38:05.939575891+05:30","message":"has retried 10 times"}

Cause

Since the Archive Tier is not supported in the selected region the headers being sent by NetBackup are not supported. Note that uploads of small files (such as the MSDP metadata files) do not set the flag so can upload successfully.

Solution

The solution is to either not use this region or to not use the Archive Tier as the backend storage.

See https://azure.microsoft.com/en-us/explore/global-infrastructure/products-by-region/?products=storage for details of which storage tiers are supported in each Azure region.