NetBackup™ Deployment Guide for Kubernetes Clusters

• Ensure that the namespace associated with Cloud Scale deployment are deleted by using the following command:

  kubectl get ns

• Confirm that storageclass, pv, clusterroles, clusterrolebindings, crd's associated with Cloud Scale deployment are deleted by using the following command:

  kubectl get sc,pv,crd,clusterrolebindings,clusterroles

For AKS

• Navigate to the snapshot resource created during backup and Create a disk under the recovered cluster infra resource group (for example, MC_<clusterRG>_<cluster name>_<cluster_region>).

• Note down the resource ID of this disk (navigate to the Properties of the disk). It can be obtained from portal/az cli.

  Format of resource ID:/subscriptions/<subscription id>/resourceGroups/<MC_<clusterRG>_<cluster name>_<cluster_region>/providers/Microsoft.Compute/disks>/<disk name>

• Create static PV using the resource ID of backed up disk. Copy the below yaml and update the pv name, size of the disk, namespace and storage class name in pgsql-pv.yaml file and apply the yaml:

  pgsql-pv.yaml

  apiVersion: v1
  kind: PersistentVolume
  metadata:
    name: <pv name>
  spec:
    capacity:
      storage: <size of the disk>
    accessModes:
      - ReadWriteOnce
    persistentVolumeReclaimPolicy: Retain
    storageClassName: <storage class name>
    claimRef:
      name: psql-pvc
      namespace: <environment namespace>
    csi:
      driver: disk.csi.azure.com
      readOnly: false
      volumeHandle: <Resorce ID of the Disk>

  Example of pgsql-pv.yaml file:

  apiVersion: v1
  kind: PersistentVolume
  metadata:
    name: psql-pv
  spec:
    capacity:
      storage: 30Gi
    accessModes:
      - ReadWriteOnce
    persistentVolumeReclaimPolicy: Retain
    storageClassName: gp2-immediate
    claimRef:
      name: psql-pvc
      namespace: nbux
    csi:
      driver: disk.csi.azure.com
      readOnly: false
      volumeHandle: /subscriptions/a332d749-22d8-48f6-9027-ff04b314e840/resourceGroups/MC_vibha-vasantraohadule-846288_auto_aks-vibha-vasantraohadule-846288_eastus2/providers/Microsoft.Compute/disks/psql-disk
  

  Create psql-pv using the following command:

  kubectl apply -f <path_to_psql_pv.yaml> -n <environment-namespace>

• Ensure that the newly created PV is in Available state before restoring the Snapshot Manager server as follows:

  kubectl get pv | grep psql-pvc

  >> psql-pv 30Gi RWO managed-premium-disk Available nbu/psql-pvc 50s

For EKS

• Navigate to the EC2 > Snapshots in AWS Console and click on the Create volume from the snapshot (expand the Actions drop down) which is taken in backup step 2 in same availability zone (AZ) of volume attached to psql-pvc (mentioned in step 1 of backup steps).

  Note down the volumeID (for example, vol-xxxxxxxxxxxxxxx).

• In case deployment is in different availability zone (AZ), user must change the availability zone (AZ) for volume and update the volumeID accordingly.

• Create static PV using the backed up volumeID. Copy the below yaml and update the pv name, size of the disk, namespace and storage class name in pgsql-pv.yaml file and apply the yaml:

  pgsql-pv.yaml

  apiVersion: v1
  kind: PersistentVolume
  metadata:
    name: <pv name>
  spec:
    accessModes:
    - ReadWriteOnce
    awsElasticBlockStore:
      fsType: <fs type>
      volumeID: <backed up volumeID>    # append this  aws://az-code/ , for e.g. aws://us-east-2b/ at the starting
    capacity:
      storage: 30Gi
    claimRef:
      apiVersion: v1
      kind: PersistentVolumeClaim
      name: psql-pvc
      namespace: <netbackup namespace>
    persistentVolumeReclaimPolicy: Retain
    storageClassName: <storage class name>
    volumeMode: Filesystem

  Sample yaml for pgsql-pv.yaml file:

  apiVersion: v1
  kind: PersistentVolume
  metadata:
    name: psql-pv
  spec:
    accessModes:
    - ReadWriteOnce
    awsElasticBlockStore:
      fsType: ext4
      volumeID: aws://us-east-2b/vol-0d86d2ca38f231ede
    capacity:
      storage: 30Gi
    claimRef:
      apiVersion: v1
      kind: PersistentVolumeClaim
      name: psql-pvc
      namespace: nbu
    persistentVolumeReclaimPolicy: Retain
    storageClassName: gp2-immediate
    volumeMode: Filesystem

  Create psql-pv using the following command:

  kubectl apply -f <path_to_psql_pv.yaml> -n <netbackup-namespace>

  kubectl get pv | grep psql-pvc

• Ensure that the newly created PV is in Available state before restoring the Snapshot Manager server as follows:

  kubectl get pv | grep psql-pvc

  >>> psql-pv 30Gi RWO gp2-immediate Available nbu/psql-pvc 50s

• Make a copy of environment CR yaml (environment_backup.yaml) file with name environment_backup_copy.yaml and save it for later use.

• Remove CP-server section from the original environment_backup.yaml file.

• Modify the environment with the paused: true field in MSDP and Media sections. Modify the following and save it:

  spec > msdpscaleout > paused to true

  spec > > mediaservers > paused to true

  Only primary server must get deployed in this case. Now apply this modified environment.yaml file using the following command:

  kubectl apply -f <environment.yaml file name>

• Once primary server is up and running:

  • Execute # kubectl exec -it -n <namespace> <primary-pod-name> -- /bin/bash command to exec into the primary pod.

  • Increase the debug logs level on primary server.

  • Create a directory DRPackages at persisted location using mkdir /mnt/nbdb/usr/openv/drpackage command and provide the permission as 757.

• Copy back the earlier copied DR files to primary pod at /mnt/nbdb/usr/openv/drpackage file using the following command:

  kubectl cp <Path_of_DRPackages_on_host_machine> <primary-pod-namespace>/<primary-pod-name>:/mnt/nbdb/usr/openv/drpackage

• Execute the following steps after executing into the primary server pod:

  • Change the ownership of files in /mnt/nbdb/usr/openv/drpackage using the chown nbsvcusr:nbsvcusr <file-name> command.

  • Deactivate NetBackup health probes using the /opt/veritas/vxapp-manage/nb-health deactivate command.

  • Stop the NetBackup services using /usr/openv/netbackup/bin/bp.kill_all command.

  • Execute the /usr/openv/netbackup/bin/admincmd/nbhostidentity -import -infile /mnt/nbdb/usr/openv/drpackage/.drpkg command.

  • Clear NetBackup host cache, run the bpclntcmd -clear_host_cache command.

  • Start NetBackup services using the /usr/openv/netbackup/bin/bp.start_all command.

  • Start NetBackup services using the /usr/openv/netbackup/bin/bp.start_all command.

  • Refresh the certificate revocation list using the /usr/openv/netbackup/bin/nbcertcmd -getcrl command.

• Run the primary server reconciler.

  This can be done by editing the environment (using kubectl edit environment -n <namespace> command) and changing primary spec's paused field to true and save it.

  Then, to enable the reconciler to run, the environment needs to be edited again and the primary's paused field in spec should again be set to false.

  The SHA fingerprint will get updated in the primary CR's status.

• Allow Auto reissue certificate from primary for MSDP, Media and Snapshot Manager server from Web UI.

  In Web UI, navigate to Security > Host Mappings > for the MSDP Storage Server, click on the 3 dots on the right > check Allow Auto reissue Certificate. Repeat this for media servers and Snapshot Manager server entries also.

• Edit the environment using the kubectl edit environment -n <namespace> command and change paused field to false for MSDP.

• Redeploy MSDP Scaleout on a cluster by using the same CR parameters and NetBackup re-issue token.

• If the LSU cloud alias does not exist, you can use the following command to add it.

  /usr/openv/netbackup/bin/admincmd/csconfig cldinstance -as -in <instance-name> -sts <storage-server-name> -lsu_name <lsu-name>

  When MSDP Scaleout is up and running, re-use the cloud LSU on NetBackup primary server.

  /usr/openv/netbackup/bin/admincmd/nbdevconfig -setconfig -storage_server <STORAGESERVERNAME> -stype PureDisk -configlist <configuration file>

  Credentials, bucket name, and sub bucket name must be the same as the recovered Cloud LSU configuration in the previous MSDP Scaleout deployment.

  Configuration file template:

  V7.5 "operation" "reuse-lsu-cloud" string
  V7.5 "lsuName" "LSUNAME" string
  V7.5 "cmsCredName" "XXX" string
  V7.5 "lsuCloudAlias" "<STORAGESERVERNAME_LSUNAME>" string
  V7.5 "lsuCloudBucketName" "XXX" string
  V7.5 "lsuCloudBucketSubName" "XXX" string
  V7.5 "lsuKmsServerName" "XXX" string

  Note:

  For Veritas Alta Recovery Vault Azure storage, the cmsCredName is a credential name and cmsCredName can be any string. Add recovery vault credential in the CMS using the NetBackup Web UI and provide the credential name for cmsCredName. For more information, see About Veritas Alta Recovery Vault Azure topic in NetBackup Deduplication Guide.

• On the first MSDP Engine of MSDP Scaleout, run the following command for each cloud LSU:

  sudo -E -u msdpsvc /usr/openv/pdde/pdcr/bin/cacontrol --catalog clouddr <LSUNAME>

• Restart the MSDP services in the MSDP Scaleout.

  Option 1: Manually delete all the MSDP engine pods.

  kubectl delete pod <sample-engine-pod> -n <sample-cr-namespace>

  Option 2: Stop MSDP services in each MSDP engine pod. MSDP service starts automatically.

  kubectl exec <sample-engine-pod> -n <sample-cr-namespace> -c uss-engine -- /usr/openv/pdde/pdconfigure/pdde stop

• For Windows: From the command prompt navigate to the agent installation directory (C:\Program Files\Veritas\CloudPoint\) and run the following command:

  #flexsnap-agent.exe --renew --token <auth_token> renew

  This command fails in the first attempt. Rerun the command for successful attempt.

• For Linux: Rerun the following command on Linux host:

  sudo flexsnap-agent --renew --token <auth_token>{}