NetBackup™ Web UI Kubernetes Administrator's Guide

Last Published:
Product(s): NetBackup & Alta Data Protection (10.1)
  1. Overview of NetBackup for Kubernetes
    1.  
      Overview
    2.  
      Features of NetBackup support for Kubernetes
  2. Deploying and configuring the NetBackup Kubernetes operator
    1.  
      Deploy service package on NetBackup Kubernetes operator
    2.  
      Port requirements for Kubernetes operator deployment
    3.  
      Upgrade the NetBackup Kubernetes operator
    4.  
      Delete the NetBackup Kubernetes operator
    5.  
      Configure NetBackup Kubernetes datamover
    6.  
      Configure settings for NetBackup snapshot operation
    7.  
      Troubleshooting NetBackup servers with short names
    8. Managing image groups
      1.  
        About image expiration
      2.  
        About image copy
  3. Deploying certificates on NetBackup Kubernetes operator
    1.  
      Deploy certificates on the Kubernetes operator
    2.  
      Perform Host-ID-based certificate operations
    3.  
      Perform ECA certificate operations
    4.  
      Identify certificate types
  4. Managing Kubernetes assets
    1.  
      Add a Kubernetes cluster
    2.  
      Configure settings
    3.  
      Add protection to the assets
  5. Managing Kubernetes intelligent groups
    1.  
      About intelligent group
    2.  
      Create an intelligent group
    3.  
      Delete an intelligent group
    4.  
      Edit an intelligent group
  6. Protecting Kubernetes assets
    1.  
      Protect an intelligent group
    2.  
      Remove protection from an intelligent group
    3.  
      Configure backup schedule
    4.  
      Configure backup options
    5.  
      Configure backups
    6.  
      Configure Auto Image Replication (AIR) and duplication
    7.  
      Configure storage units
  7. Recovering Kubernetes assets
    1.  
      Explore and validate recovery points
    2.  
      Restore from snapshot
    3.  
      Restore from backup copy
  8. Troubleshooting Kubernetes issues
    1.  
      Error during the primary server upgrade: NBCheck fails
    2.  
      Error during an old image restore: Operation fails
    3.  
      Error during persistent volume recovery API
    4.  
      Error during restore: Final job status shows partial failure
    5.  
      Error during restore on the same namespace
    6.  
      Datamover pods exceed the Kubernetes resource limit
    7.  
      Error during restore: Job fails on the highly loaded cluster
    8.  
      Custom Kubernetes role created for specific clusters cannot view the jobs
    9.  
      Openshift creates blank non-selected PVCs while restoring applications installed from OperatorHub

Configure settings for NetBackup snapshot operation

You need to configure snapshot operation on the Kubernetes operator deployment before you perform the actual backup from snapshot operations.

  1. Define a storage class pointing to the CSI plugin.

  2. Define a VolumeSnapshotClass class consisting of CSI driver details.

  3. Label the volume snapshot class for NetBackup usage. Add the following label netbackup.veritas.com/default-csi-volume-snapshot-class=true.

    Note:

    Snapshot of a namespace consisting of persistent volume fails with an error message: Failed to create snapshot of the Kubernetes namespace.

    The snapshot operation may fail due to multiple reasons, for example, a valid volume snapshot class for the driver with label volumesnapshotclass is not found.

  4. Sizing for metadata persistent volume is required. The default persistent volume size for Kubernetes operator is 10Gi. The persistent volume size is configurable.

    You can change the value for storage from 10Gi to a higher value before deploying the plugin. This leads to the nbukops pod have the size of the PVC mounted in the pod.

    Persistent Volume Claim looks like this:

    apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      labels:
        component: netbackup
      name: {{ .Release.Namespace }}-netbackupkops
      namespace: {{ .Release.Namespace }}
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 10Gi
    
    • During fresh installation while configuring the Helm Chart. You can modify the size of PVC storage in the deployment.yaml of the netbackupkops-helm-chart which leads to creation of the initial PVC size.

    • Post installation, updating the PVC size (dynamic volume expansion) is supported by few storage vendors. For more information, refer tohttps://kubernetes.io/docs/concepts/storage/persistent-volumes

Note:

The default size of persistent volume can be resized to larger value without losing the data. You are recommended to add the storage provider that supports volume expansion.

Note:

To get the configuration value, you can run the command: kubectl get configmaps <namespace>-backup-operator-configuration -n <namespace> -o yaml > {local.file}

Table: Kubernetes operators supported configuration parameters in <namespace>-backup-operator-configuration

Configuration

Description

Default value

Possible value

DaemonSets

A Daemonset is a dynamic object in Kubernetes which is managed by a controller. You can set the desired state that represents the specific pods that need to exist on every node. The pod compromise in the control loop can compare the current practical state with the desired state.

true

true, false

Deployments

Deployments for the Kubernetes workload.

true

true, false

Pods

A pod is the smallest execution unit in Kubernetes.

true

true, false

ReplicaSets

Replica Set ensures how many replicas of a pod should be running. It can be considered as a replacement of the replication controller.

true

true, false

Secrets

Secrets are the objects that contain sensitive data such as passwords, tokens, and credentials.

true

true, false

Services

Services offered in Kubernetes.

true

true, false

namespace

Kubernetes operator is deployed in the namespace.

Any name given to a namespace.

NetBackup namespace.

cleanStaleCRDurationMinutes

Time duration after a CR job is invoked to clean stale CRs. The interval after which stale custom resource cleanup job is triggered.

24 hours

1440 minutes

ttlCRDurationMinutes

TTL CR duration

minutes

30240 minutes

livesnessProbeInitialDelay

Probe initial delay period.

minutes

60 minutes

livenessProbePeriodSeconds

Probe period.

seconds

80 seconds

checkNbcertdaemonStatusDurationMinutes

NB certificate daemon status duration.

minutes

1440 minutes

collectDataMoverLogs

Due to high memory usage in datamover logs collection, it is recommended to enable the logs only when you are debugging, troubleshooting, or restarting the pods.

Before enabling the logs for datamover, ensure to increase the memory limits for NetBackup Kubernetes pod to at least 2 GB or more. After the debugging or troubleshooting is done, you can reset to the previous or the default value.

Note:

Granular support is provided for collecting datamover logs only in case of failed jobs. It provides an additional level of granularity layer, All/FailedOnly/Off.

Failed

All, Failed, None

maxRetentionDataMoverLogsInHours

Maximum retention for datamover logs.

24 hours

72 hours

maxRetentionDataMoverInHours

It removes all the datamover resources that are older than the specified time.

24 hours

24 hours

cleanStaleCertFilesDurationMinutes

The interval after which stale certificate files cleanup job is triggered.

60 minutes

1440 minutes

maxRetentionInDiscoveryCacheHours

It is the time in hours that decides the time interval for keeping the discovery cache.

24 hours

48 hours

pollingTimeoutInMinutes

It is the timeout that keeps retrying till it expires and fails.

15 minutes

15 minutes

pollingFrequencyInSecs

Polling frequency.

seconds

5 seconds

nbcertPrerequisteDirectoryAndFiles

NBCA prerequisites.

Certificate name

Certificate name

Prerequisites for backup from snapshot and restore from backup operations
  1. Label a valid storage class for NetBackup usage, add the following label: netbackup.veritas.com/default-csi-storage-class=true. If NetBackup labeled storage class is not found, then backup from snapshot job for metadata image and restore jobs fail with the error message No eligible storage classes found.

    To label the storage classes, run the following commands given in the examples:

    Example 1. Run the command:# oc get sc

    Name

    Provisioner

    ocs-storagecluster-ceph-rbd (default)

    openshift-storage.rbd.csi.ceph.com

    ocs-storagecluster-ceph-rgw

    openshift-storage.ceph.rook.io/bucket

    ocs-storagecluster-cephfs

    openshift-storage.cephfs.csi.ceph.com

    openshift-storage.noobaa.io

    openshift-storage.noobaa.io/obc

    thin

    kubernetes.io/vsphere-volume

    Reclaim policy

    Volume binding mode

    Allow volume expansion

    Age

    Delete

    Immediate

    True

    2d2h

    Delete

    Immediate

    False

    2d2h

    Delete

    Immediate

    True

    2d2h

    Delete

    Immediate

    False

    2d2h

    Delete

    Immediate

    False

    19h

    Example 2. Run the command:# oc get sc ocs-storagecluster-ceph-rbd --show-labels

    Name

    Provisioner

    Reclaim policy

    ocs-storagecluster-ceph-rbd (default)

    openshift-storage.rbd.csi.ceph.com

    Delete

    Volume binding mode

    Allow volume expansion

    Age

    Label

    Immediate

    True

    2d2h

    netbackup.veritas.com/default-csi-storage-class=true

    Example 3. Run the command:# oc label sc ocs-storagecluster-cephfs netbackup.veritas.com/default-csi-storage-class=true

    storageclass.storage.k8s.io/ocs-storagecluster-cephfs labeled

    Example 4. Run the command:oc get sc ocs-storagecluster-cephfs --show-labels

    Name

    Provisioner

    Reclaim policy

    ocs-storagecluster-cephfs

    openshift-storage.cephfs.csi.ceph.com 

    Delete

    Volume binding mode

    Allow volume expansion

    Age

    Label

    Immediate

    True

    2d2h

    netbackup.veritas.com/default-csi-storage-class=true

  2. Label a valid volume snapshot class for NetBackup usage, add the following label: netbackup.veritas.com/default-csi-volume-snapshot-class=true. If the NetBackup labeled VolumeSnapshotClass class is not found, then backup from snapshot job for metadata image and restore jobs fails with an error message: Failed to create snapshot of the Kubernetes namespace.

    To label the volume snapshot classes, run the following commands given the examples:

    Example 1. Run the command:# oc get volumesnapshotclass

    Name

    Driver

    ocs-storagecluster-cephfsplugin-snapclass

    openshift-storage.cephfs.csi.ceph.com

    ocs-storagecluster-rbdplugin-snapclass

    openshift-storage.rbd.csi.ceph.co

    Deletion policy

    Age

    Delete

    2d2h

    Delete

    2d2h

    Example 2. Run the command:# oc get volumesnapshotclass ocs-storagecluster-cephfsplugin-snapclass --show-labels

    Name

    Driver

    ocs-storagecluster-cephfsplugin-snapclass

    openshift-storage.cephfs.csi.ceph.com

    Deletion policy

    Age

    Delete

    2d2h

    Example 3. Run the command:# oc label volumesnapshotclass ocs-storagecluster-cephfsplugin-snapclass netbackup.veritas.com/default-csi-volume-snapshot-class=true

    volumesnapshotclass.snapshot.storage.k8s.io/ocs-storagecluster-cephfsplugin-snapclass labeled

    Example 4. Run the command:# oc get volumesnapshotclass ocs-storagecluster-cephfsplugin-snapclass --show-labels

    Name

    Driver

    ocs-storagecluster-cephfsplugin-snapclass

    openshift-storage.cephfs.csi.ceph.com

    Deletion policy

    Age

    Labels

    Delete

    2d2h

    netbackup.veritas.com/default-csi-volume-snapshot-class=true

  3. Each primary server which runs the backup from snapshot and restore from backup copy operations, needs to create a separate ConfigMap with the primary server's name.

    In the following configmap.yaml example,

    • backupserver.sample.domain.com and mediaserver.sample.domain.com are the hostnames of NetBackup primary and media server.

    • IP: 10.20.12.13 and IP: 10.21.12.13 are the IP addresses of NetBackup primary and media server.

    apiVersion: v1
    data:
      datamover.hostaliases: |
            10.20.12.13=backupserver.sample.domain.com
            10.21.12.13=mediaserver.sample.domain.com
      datamover.properties: |
            image=reg.domain.com/datamover/image:latest
      version: "1"
    kind: ConfigMap
    metadata:
      name: backupserver.sample.domain.com
      namespace: kops-ns
    
    • Copy the configmap.yaml file details.

    • Open the text editor and paste the yaml file details.

    • Then, save it with the yaml file extension to the home directory from where the Kubernetes clusters are accessible.

  4. Specify datamover.properties: image=reg.domain.com/datamover/image:latest with correct datamover image.

  5. Specify datamover.hostaliases, if the primary server and the media servers connected to the primary server have short names and host resolution failing from datamover. Provide a mapping of all hostnames to IPs for primary and media servers.

  6. Create a secret as described in detail in the Point 6 in the Deploy service package on NetBackup Kubernetes operator section to use a private docker registry.

    Once the secret is created, add the following attributes while creating a configmap.yaml file.

    • datamover.properties: |

    • image=repo.azurecr.io/netbackup/datamover:10.0.0049

    • imagePullSecret=secret_name

  7. Create the configmap.yaml file, run the command: kubectl create -f configmap.yaml.

  8. If Kubernetes operator is not able to resolve the primary server based on short names

    • While fetching the certificates, if you get a message:EXIT STATUS 8500: Connection with the web service was not established. Then, verify the hostname resolution state from the nbcert logs.

    • If the hostname resolution fails, then do the following:

      Update the kops deployment.yaml and add the hostAliases in the deployment.

    • In the following hostAliases example,

      • backupserver.sample.domain.com and mediaserver.sample.domain.com are the hostnames of NetBackup primary and media server.

      • IP: 10.20.12.13 and IP: 10.21.12.13 are the IP addresses of NetBackup primary and media server.

      hostAliases:
      - hostnames:
        - backupserver.sample.domain.com
        ip: 10.20.12.13
      - hostnames:
        -  mediaserver.sample.domain.com
        ip: 10.21.12.13
      

      Copy, paste the hostAliases example details in the text editor and add to the hostAliases in the deployment.

      Note:

      The hostAliases section must be added to the line 2104 in default ./netbackupkops-helm-chart/templates/deployment.yaml file.

      hostAliases example:

      2104	hostAliases;
      -	ip:10.15.206.7
      hostnames:
      -	lab02-linsvr-01.demo.sample.domain.com
      -	lab02-linsvr-01
      -	ip:10.15.206.8
      hostnames:
      -	lab02-linsvr-02.demo.sample.domain.com
      -	lab02-linsvr-02
      imagePullSecrets:
      -	name:  {{ .values.netbackupKops.imagePullSecrets.name}}
      
  9. Create a secret with fingerprint and authorization token.

  10. Create a backupservercert request to fetch certificates.

    For more information, refer to the NetBackup™ Security and Encryption Guide.

DTE client settings supported in Kubernetes workload

The DTE_CLIENT_MODE option specifies the data-in-transit encryption (DTE) mode that is set on the datamover via backupserver specific configmap. Data-in-transit encryption of backup images is carried out based on the global DTE mode and the client DTE mode.

Update the backupserver specific configmap and add DTE_CLIENT_MODE key to it. This key can take following values:

  • AUTOMATIC

  • ON

  • OFF

For more information on the DTE_CLIENT_MODE, refer to the DTE_CLIENT_MODE for clients section in the Veritas NetBackup™ Administrator's Guide, Volume I .

Following is the configmap with DTE_CLIENT_MODE setting added:

apiVersion: v1
data:
  datamover.hostaliases: |
        10.20.12.13=backupserver.sample.domain.com
        10.21.12.13=mediaserver.sample.domain.com
  datamover.properties: |
        image=reg.domain.com/datamover/image:latest
        DTE_CLIENT_MODE=ON
  version: "1"
kind: ConfigMap
metadata:
  name: backupserver.sample.domain.com
  namespace: kops-ns
Customization of datamover properties

You can customize datamover properties by passing key-value pairs in the backup server specific configmap.

Table: Datamover properties

Key Name

Possible Values

VXMS_VERBOSE

Range:[0,99]

VERBOSE

Range:[0,5]

DTE_CLIENT_MODE

  • AUTOMATIC

  • ON

  • OFF

To update the configmap, add the key value pairs as follows:

apiVersion: v1
data:
  datamover.properties: |
        image=reg.domain.com/datamover/image:latest
        VERBOSE=5
        DTE_CLIENT_MODE=OFF
        VXMS_VERBOSE=5
  version: "1"
kind: ConfigMap
metadata:
  name: backupserver.sample.domain.com
  namespace: kops-ns