NetBackup™ Deployment Guide for Kubernetes Clusters

• Ensure that the namespace associated with Cloud Scale deployment are deleted by using the following command:

  kubectl get ns

• Confirm that storageclass, pv, clusterroles, clusterrolebindings, crd's associated with Cloud Scale deployment are deleted by using the following command:

  kubectl get sc,pv,crd,clusterrolebindings,clusterroles

• Deploy the dbtrust.yaml file as follows:

  Create dbtrust.yaml file and add below to it:

  apiVersion: trust.cert-manager.io/v1alpha1
  kind: Bundle
  metadata:
    name: db-cert
    namespace: netbackup
  spec:
    sources:
    - secret:
        name: "postgresql-netbackup-ca"
        key: "tls.crt"
    target:
      namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: netbackup
      configMap:
        key: "dbcertpem"

  Run the following command:

  kubectl apply -f dbtrust.yaml

For AKS

• Navigate to the snapshot resource created during backup and Create a disk under the recovered cluster infra resource group (for example, MC_<clusterRG>_<cluster name>_<cluster_region>).

• Note down the resource ID of this disk (navigate to the Properties of the disk). It can be obtained from portal/az cli.

  Format of resource ID:/subscriptions/<subscription id>/resourceGroups/<MC_<clusterRG>_<cluster name>_<cluster_region>/providers/Microsoft.Compute/disks>/<disk name>

• Create static PV using the resource ID of backed up disk. Copy the below yaml and update the pv name, size of the disk, namespace and storage class name in pgsql-pv.yaml file and apply the yaml:

  pgsql-pv.yaml

  apiVersion: v1
  kind: PersistentVolume
  metadata:
    name: <pv name>
  spec:
    capacity:
      storage: <size of the disk>
    accessModes:
      - ReadWriteOnce
    persistentVolumeReclaimPolicy: Retain
    storageClassName: <storage class name>
    claimRef:
      name: psql-pvc
      namespace: <environment namespace>
    csi:
      driver: disk.csi.azure.com
      readOnly: false
      volumeHandle: <Resorce ID of the Disk>

  Example of pgsql-pv.yaml file:

  apiVersion: v1
  kind: PersistentVolume
  metadata:
    name: psql-pv
  spec:
    capacity:
      storage: 30Gi
    accessModes:
      - ReadWriteOnce
    persistentVolumeReclaimPolicy: Retain
    storageClassName: gp2-immediate
    claimRef:
      name: psql-pvc
      namespace: nbux
    csi:
      driver: disk.csi.azure.com
      readOnly: false
      volumeHandle: /subscriptions/a332d749-22d8-48f6-9027-ff04b314e840/resourceGroups/MC_vibha-vasantraohadule-846288_auto_aks-vibha-vasantraohadule-846288_eastus2/providers/Microsoft.Compute/disks/psql-disk
  

  Create psql-pv using the following command:

  kubectl apply -f <path_to_psql_pv.yaml> -n <environment-namespace>

• Ensure that the newly created PV is in Available state before restoring the Snapshot Manager server as follows:

  kubectl get pv | grep psql-pvc

  >> psql-pv 30Gi RWO managed-premium-disk Available nbu/psql-pvc 50s

For EKS

• Navigate to the EC2 > Snapshots in AWS Console and click on the Create volume from the snapshot (expand the Actions drop down) which is taken in backup step 2 in same availability zone (AZ) of volume attached to psql-pvc (mentioned in step 1 of backup steps).

  Note down the volumeID (for example, vol-xxxxxxxxxxxxxxx).

• In case deployment is in different availability zone (AZ), user must change the availability zone (AZ) for volume and update the volumeID accordingly.

• Create static PV using the backed up volumeID. Copy the below yaml and update the pv name, size of the disk, namespace and storage class name in pgsql-pv.yaml file and apply the yaml:

  pgsql-pv.yaml

  apiVersion: v1
  kind: PersistentVolume
  metadata:
    name: <pv name>
  spec:
    accessModes:
    - ReadWriteOnce
    awsElasticBlockStore:
      fsType: <fs type>
      volumeID: <backed up volumeID>    # append this  aws://az-code/ , for e.g. aws://us-east-2b/ at the starting
    capacity:
      storage: 30Gi
    claimRef:
      apiVersion: v1
      kind: PersistentVolumeClaim
      name: psql-pvc
      namespace: <netbackup namespace>
    persistentVolumeReclaimPolicy: Retain
    storageClassName: <storage class name>
    volumeMode: Filesystem

  Sample yaml for pgsql-pv.yaml file:

  apiVersion: v1
  kind: PersistentVolume
  metadata:
    name: psql-pv
  spec:
    accessModes:
    - ReadWriteOnce
    awsElasticBlockStore:
      fsType: ext4
      volumeID: aws://us-east-2b/vol-0d86d2ca38f231ede
    capacity:
      storage: 30Gi
    claimRef:
      apiVersion: v1
      kind: PersistentVolumeClaim
      name: psql-pvc
      namespace: nbu
    persistentVolumeReclaimPolicy: Retain
    storageClassName: gp2-immediate
    volumeMode: Filesystem

  Create psql-pv using the following command:

  kubectl apply -f <path_to_psql_pv.yaml> -n <netbackup-namespace>

  kubectl get pv | grep psql-pvc

• Ensure that the newly created PV is in Available state before restoring the Snapshot Manager server as follows:

  kubectl get pv | grep psql-pvc

  >>> psql-pv 30Gi RWO gp2-immediate Available nbu/psql-pvc 50s

• Make a copy of environment CR yaml (environment_backup.yaml) file with name environment_backup_copy.yaml and save it for later use.

• Remove CP-server section from the original environment_backup.yaml file.

• Modify the environment with the paused: true field in MSDP and Media sections. Modify the following and save it:

  spec > msdpscaleout > paused to true

  spec > > mediaservers > paused to true

  Only primary server must get deployed in this case. Now apply this modified environment.yaml file using the following command:

  kubectl apply -f <environment.yaml file name>

• Once primary server is up and running:

  • Perform the following steps for NBATD pod recovery:

    • Create the DRPackages directory on persisted location /mnt/nblogs/ in nbatd pod by executing the following command:

      kubectl exec -it -n <namespace> <nbatd-pod-name> --/bin/bash

      mkdir /mnt/nblogs/DRPackages

    • Copy DR files which were saved when performing DR backup to nbatd pod at /mnt/nblogs/DRPackages using the following command:

      kubectl cp <Path_of_DRPackages_on_host_machine> <nbatd-pod-namespace>/<nbatd-pod-name>:/mnt/nblogs/DRPackages

    • Execute the following steps in the nbatd pod:

      • Execute the kubectl exec -it -n <namespace> <nbatd-pod-name> --/bin/bash command.

      • Deactivate nbatd health probes using the /opt/veritas/vxapp-manage/nbatd_health.sh disable command.

      • Stop the nbatd service using /opt/veritas/vxapp-manage/nbatd_stop.sh 0 command.

      • Execute the /opt/veritas/vxapp-manage/nbatd_identity_restore.sh -infile /mnt/nblogs/DRPackages/ (DR package name) command.

  • Execute # kubectl exec -it -n <namespace> <primary-pod-name> -- /bin/bash command to exec into the primary pod.

  • Increase the debug logs level on primary server.

  • Create a directory DRPackages at persisted location using mkdir /mnt/nbdb/usr/openv/drpackage command and provide the permission as 757.

• Copy back the earlier copied DR files to primary pod at /mnt/nbdb/usr/openv/drpackage file using the following command:

  kubectl cp <Path_of_DRPackages_on_host_machine> <primary-pod-namespace>/<primary-pod-name>:/mnt/nbdb/usr/openv/drpackage

• Execute the following steps after executing into the primary server pod:

  • Change the ownership of files in /mnt/nbdb/usr/openv/drpackage using the chown nbsvcusr:nbsvcusr <file-name> command.

  • Deactivate NetBackup health probes using the /opt/veritas/vxapp-manage/nb-health deactivate command.

  • Stop the NetBackup services using /usr/openv/netbackup/bin/bp.kill_all command.

  • Execute the /usr/openv/netbackup/bin/admincmd/nbhostidentity -import -infile /mnt/nbdb/usr/openv/drpackage/.drpkg command.

  • Clear NetBackup host cache, run the bpclntcmd -clear_host_cache command.

  • Restart the pods as follows:

    • Navigate to the VRTSk8s-netbackup-<version>/scripts folder.

    • Run the cloudscale_restart.sh script with Restart option as follows:

      ./cloudscale_restart.sh <action> <namespace>

      Provide the namespace and the required action:

      stop: Stops all the services under primary server (waits until all the services are stopped).

      start: Starts all the services and waits until the services are up and running under primary server.

      restart: Stops the services and waits until all the services are down. Then starts all the services and waits until the services are up and running.

    Note:

    Ignore if policy job pod does not come up in running state. Policy job pod would start once primary services start.

  • Refresh the certificate revocation list using the /usr/openv/netbackup/bin/nbcertcmd -getcrl command.

• Run the primary server reconciler.

  This can be done by editing the environment (using kubectl edit environment -n <namespace> command) and changing primary spec's paused field to true and save it.

  Then, to enable the reconciler to run, the environment needs to be edited again and the primary's paused field in spec should again be set to false.

  The SHA fingerprint will get updated in the primary CR's status.

• Allow Auto reissue certificate from primary for MSDP, Media and Snapshot Manager server from Web UI.

  In Web UI, navigate to Security > Host Mappings > for the MSDP Storage Server, click on the 3 dots on the right > check Allow Auto reissue Certificate. Repeat this for media servers and Snapshot Manager server entries also.

• Edit the environment using the kubectl edit environment -n <namespace> command and change paused field to false for MSDP.

• Redeploy MSDP Scaleout on a cluster by using the same CR parameters and NetBackup re-issue token.

• If the LSU cloud alias does not exist, you can use the following command to add it.

  /usr/openv/netbackup/bin/admincmd/csconfig cldinstance -as -in <instance-name> -sts <storage-server-name> -lsu_name <lsu-name>

  When MSDP Scaleout is up and running, re-use the cloud LSU on NetBackup primary server.

  /usr/openv/netbackup/bin/admincmd/nbdevconfig -setconfig -storage_server <STORAGESERVERNAME> -stype PureDisk -configlist <configuration file>

  Credentials, bucket name, and sub bucket name must be the same as the recovered Cloud LSU configuration in the previous MSDP Scaleout deployment.

  Configuration file template:

  V7.5 "operation" "reuse-lsu-cloud" string
  V7.5 "lsuName" "LSUNAME" string
  V7.5 "cmsCredName" "XXX" string
  V7.5 "lsuCloudAlias" "<STORAGESERVERNAME_LSUNAME>" string
  V7.5 "lsuCloudBucketName" "XXX" string
  V7.5 "lsuCloudBucketSubName" "XXX" string
  V7.5 "lsuKmsServerName" "XXX" string

  Note:

  For Veritas Alta Recovery Vault Azure storage, the cmsCredName is a credential name and cmsCredName can be any string. Add recovery vault credential in the CMS using the NetBackup Web UI and provide the credential name for cmsCredName. For more information, see About Veritas Alta Recovery Vault Azure topic in NetBackup Deduplication Guide.

• On the first MSDP Engine of MSDP Scaleout, run the following command for each cloud LSU:

  sudo -E -u msdpsvc /usr/openv/pdde/pdcr/bin/cacontrol --catalog clouddr <LSUNAME>

• Restart the MSDP services in the MSDP Scaleout.

  Option 1: Manually delete all the MSDP engine pods.

  kubectl delete pod <sample-engine-pod> -n <sample-cr-namespace>

  Option 2: Stop MSDP services in each MSDP engine pod. MSDP service starts automatically.

  kubectl exec <sample-engine-pod> -n <sample-cr-namespace> -c uss-engine -- /usr/openv/pdde/pdconfigure/pdde stop

• Navigate to the VRTSk8s-netbackup-<version>/scripts folder.

• Run the cloudscale_restart.sh script with Restart option as follows:

  ./cloudscale_restart.sh <action> <namespace>

  Provide the namespace and the required action:

  stop: Stops all the services under primary server (waits until all the services are stopped).

  start: Starts all the services and waits until the services are up and running under primary server.

  restart: Stops the services and waits until all the services are down. Then starts all the services and waits until the services are up and running.

• If on host agent fails, run the following respective commands:

  • For Windows: From the command prompt navigate to the agent installation directory (C:\Program Files\Veritas\CloudPoint\) and run the following command:

    #flexsnap-agent.exe --renew --token <auth_token> renew

    This command fails in the first attempt. Rerun the command for successful attempt.

  • For Linux: Rerun the following command on Linux host:

    sudo flexsnap-agent --renew --token <auth_token>{}

• After Snapshot Manager recovery, if some SLP jobs are failing repetitively due to some pending operations before disaster recovery, then cancel the Storage Lifecycle Policy (SLP) jobs using the nbstlutil command.

  For more information on the nbstlutil command, refer to the NetBackup™ Commands Reference Guide.