Description

Technical Document: Resolving Public vs Private Endpoint Connectivity for NetBackup MSDP-C to Google Cloud Storage

1. Background and Problem Statement

The customer intended to route NetBackup Media Server Deduplication Pool Cloud (MSDP-C) traffic to Google Cloud Storage (GCS) via a private endpoint. However, data transfers were observed to be using the public internet instead of the internal/private network, despite host file and configuration changes. The customer required all backup and duplication traffic to use the private Google endpoint for security and performance reasons.

2. Symptoms and Observations

• Traffic was routed via public IPs even though private endpoints and host file entries were configured.
• Disk volumes were DOWN for cloud storage units after moving to the private endpoint.
• SSL certificate errors (hostname mismatch) were observed when connecting to the bucket FQDN.
• Alias misconfiguration: Aliases intended for the private endpoint were still present under the public instance.
• CRL (Certificate Revocation List) checks were enabled, causing issues with volume status.
• Incomplete certificate chain in cacert.pem on primary and media servers.
• Name resolution errors for the bucket FQDNs.
• NetBackup services (pdde) required restart after configuration changes.
  
  Errors Seen
  
  OCSD LOG: 
  
  {"level":"error","errmsg":"RequestError: send request failed\ncaused by: Head \"https://customhostname.p.googleapis.com/Stat/Test/Not/Exist/Object\": dial tcp: lookup <customere curl>.p.googleapis.com on <private ip>: no such host","storageConfigID":"StorageID","Alias":"<custom alias>","time":"2025-08-20T22:06:28.54559831+03:00","caller":"main.ensureStorageConfigInCacheAllowBucketEmpty","message":"[ensureStorageConfigInCache] Unable to load storage configuration. Removed it from storageConfigMap"}
  
  "level":"error","errmsg":"RequestError: send request failed\ncaused by: Head \"https://<customer host name>.p.googleapis.com/Stat/Test/Not/Exist/Object\": tls: fa for *.google.com, *.appengine.google.com, *.bdn.dev, *.origin-test.bdn.dev, *.cloud.google.com, *.crowdsource.google.com, *.datacompute.google.com, *.google.ca, *.google.cl, *.google.co.in, *.google.co.jp, *.google.co.uk, *.google.com.ar, *.google.com.au, *.google.com.br, *.google.com.co, *.google.com.mx, *.google.com.tr, *.google.com.vn, *.google.de, *.google.es, *.google.fr, *.google.hu, *.google.it, *.google.nl, *.google.pl, *.google.pt, *.googleapis.cn, *.googlevideo.com, *.gstatic.cn, *.gstatic-cn.com, googlecnapps.cn, *.googlecnapps.cn, googleapps-cn.com, *.googleapps-cn.com, gkecnapps.cn, *.gkecnapps.cn, googledownloads.cn, *.googledownloads.cn, recaptcha.net.cn, *.recaptcha.net.cn, recaptcha-cn.net, *.recaptcha-cn.net, widevine.cn, *.widevine.cn, ampproject.org.cn, *.ampproject.org.cn, ampproject.net.cn, *.ampproject.net.cn, google-analytics-cn.com, *.google-analytics-cn.com, googleadservices-cn.com, *.googleadservices-cn.com, googlevads-cn.com, *.googlevads-cn.com, googleapis-cn.com, *.googleapis-cn.com, googleoptimize-cn.com, *.googleoptimize-cn.com, doubleclick-cn.net, *.doubleclick-cn.net, *.fls.doubleclick-cn.net, *.g.doubleclick-cn.net, doubleclick.cn, *.doubleclick.cn, *.fls.doubleclick.cn, *.g.doubleclick.cn, dartsearch-cn.net, *.dartsearch-cn.net, googletraveladservices-cn.com, *.googletraveladservices-cn.com, googletagservices-cn.com, *.googletagservices-cn.com, googletagmanager-cn.com, *.googletagmanager-cn.com, googlesyndication-cn.com, *.googlesyndication-cn.com, *.safeframe.googlesyndication-cn.com, app-measurement-cn.com, *.app-measurement-cn.com, gvt1-cn.com, *.gvt1-cn.com, gvt2-cn.com, *.gvt2-cn.com, 2mdn-cn.net, *.2mdn-cn.net, googleflights-cn.net, *.googleflights-cn.net, admob-cn.com, *.admob-cn.com, googlesandbox-cn.com, *.googlesandbox-cn.com, *.safenup.googlesandbox-cn.com, *.gstatic.com, *.metric.gstatic.com, *.gvt1.com, *.gcpcdn.gvt1.com, *.gvt2.com, *.gcp.gvt2.com, *.url.google.com, *.youtube-nocookie.com, *.ytimg.com, ai.android, android.com, *.android.com, *.flash.android.com, g.cn, *.g.cn, g.co, *.g.co, goo.gl, www.goo.gl, google-analytics.com, *.google-analytics.com, google.com, googlecommerce.com, *.googlecommerce.com, ggpht.cn, *.ggpht.cn, urchin.com, *.urchin.com, youtu.be, youtube.com, *.youtube.com, music.youtube.com, *.music.youtube.com, youtubeeducation.com, *.youtubeeducation.com, youtubekids.com, *.youtubekids.com, yt.be, *.yt.be, android.clients.google.com, *.android.google.cn, *.chrome.google.cn, *.developers.google.cn, *.aistudio.google.com, not <customer hostname>.p.googleapis.com","status code":1013,"time":"2025-08-20T22:08:19.941382508+03:00","caller":"main.(*OCSS3).Init","message":"Unable to HeadObject for verify purpose"}

3. Root Cause Analysis

3.1. Alias and Instance Misconfiguration

• Aliases for storage servers were incorrectly associated with the public Google instance (google.com) instead of the custom private instance (private-google.com).
• Storage class mismatches between public and private instances (e.g., "Archive" vs "Standard").

3.2. Certificate and CRL Issues

• The certificate chain in cacert.pem was incomplete, missing the last lines, leading to SSL handshake failures.
• CRL checks were enabled, which was not compatible with the private endpoint configuration.

3.3. DNS and Host File Issues

• The bucket FQDNs did not resolve to the private IPs due to missing or incorrect /etc/hosts entries.

3.4. Outdated CloudProvider Package

• The system was using an outdated CloudProvider.xml package (v2.11.0), while a newer version (v2.13.6) was available and required for custom instance support.

4. Resolution Steps

4.1. Update CloudProvider Package

• Download the latest CloudProvider.xml package from Veritas Support.
• Place the file in /usr/openv/var/global/cloud and set ownership to the NetBackup service user (e.g., nbuservice:bin).
• Run csconfig r to reload the configuration.

4.2. Create and Configure Custom Private Instance

• Create a new custom instance for the private endpoint:

  /usr/openv/netbackup/bin/admincmd/csconfig cldinstance -a -in private-google.com -pt google -sh <Private host>.p.googleapis.com

   

• Add region and location constraints if required:

  /usr/openv/netbackup/bin/admincmd/csconfig cldinstance -ar -in private-google.com -lc europe-west2 -rn <region_name> -sh <privatehost>.p.googleapis.com

   

4.3. Remove Aliases from Public Instance

• Remove all storage server aliases from the public instance:

  /usr/openv/netbackup/bin/admincmd/csconfig cldinstance -rs -in google.com -sts <alias_name>

   

4.4. Add Aliases to Private Instance

• Add each storage server as an alias to the private instance with the correct storage class and SSL/CRL settings:

  /usr/openv/netbackup/bin/admincmd/csconfig cldinstance -as -in private-google.com -sts <storage_server> -lsu_name <lsu_name> -storage_class ARCHIVE -ssl 2 -crl 0

   

  • Example:

    /usr/openv/netbackup/bin/admincmd/csconfig cldinstance -as -in private-google.com -sts <STS NAME>  -storage_class ARCHIVE -ssl 2 -crl 0

     

4.5. Update Certificate Chain

• Use OpenSSL to retrieve the full certificate chain:

  openssl s_client -showcerts -connect <Privatehost>.p.googleapis.com:443 | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' | tee s3-cert-chain

   

• Ensure the full chain is present in cacert.pem on both primary and media servers. Append if necessary.

4.6. Update /etc/hosts

• Add entries for each bucket FQDN to resolve to the private IP:

  <private host ip> <endpoint address>.p.googleapis.com

   

4.7. Disable CRL Checks

• Disable CRL for all aliases and the instance:

  /usr/openv/netbackup/bin/admincmd/csconfig cldinstance -us -in private-google.com -sts <alias_name> -crl 0

   

4.8. Restart NetBackup Deduplication Services

• Restart pdde services on all affected media servers for changes to take effect.

5. Verification

• Confirm all disk volumes are UP:

  /usr/openv/netbackup/bin/admincmd/nbdevquery -listdv -stype PureDisk -dp <disk_pool_name> -U

   

• Run test backup and duplication jobs to verify data flows through the private endpoint.
• Use network monitoring tools or logs to confirm traffic is not traversing the public internet.

6. Key Lessons and Best Practices

• Always remove aliases from the public instance when moving to a private endpoint to avoid fallback to public routing.
• Ensure certificate chains are complete and match the endpoint FQDNs.
• CRL checks may need to be disabled for private endpoint configurations.
• Update CloudProvider.xml to the latest version to support custom/private instances.
• Restart services after making configuration changes.
• Validate with test jobs and network monitoring to confirm private endpoint usage.

7. References (from Case Notes Only)

• Veritas CloudProvider.xml Download
• Veritas Technote: How to adjust bandwidth throttling for msdp cloud tiering
• Case logs and commands as detailed in the case notes above.

8. Summary Table: Plan of Action

9. Conclusion

By following the above plan, the customer successfully routed all NetBackup MSDP-C traffic to Google Cloud Storage via the private endpoint, ensuring secure and efficient data transfer. All steps and troubleshooting were derived strictly from the provided case notes and internal communications.