NetBackup™ Deployment Guide for Kubernetes Clusters
- Introduction
- Section I. Configurations
- Prerequisites
- Recommendations and Limitations
- Configurations
- Configuration of key parameters in Cloud Scale deployments
- Tuning touch files
- Setting maximum jobs per client
- Setting maximum jobs per media server
- Enabling intelligent catalog archiving
- Enabling security settings
- Configuring email server
- Reducing catalog storage management
- Configuring zone redundancy
- Enabling client-side deduplication capabilities
- Parameters for logging (fluentbit)
- Section II. Deployment
- Section III. Monitoring and Management
- Monitoring NetBackup
- Monitoring Snapshot Manager
- Monitoring fluentbit
- Monitoring MSDP Scaleout
- Managing NetBackup
- Managing the Load Balancer service
- Managing PostrgreSQL DBaaS
- Managing fluentbit
- Performing catalog backup and recovery
- Section IV. Maintenance
- PostgreSQL DBaaS Maintenance
- Patching mechanism for primary, media servers, fluentbit pods, and postgres pods
- Upgrading
- Cloud Scale Disaster Recovery
- Uninstalling
- Troubleshooting
- Troubleshooting AKS and EKS issues
- View the list of operator resources
- View the list of product resources
- View operator logs
- View primary logs
- Socket connection failure
- Resolving an issue where external IP address is not assigned to a NetBackup server's load balancer services
- Resolving the issue where the NetBackup server pod is not scheduled for long time
- Resolving an issue where the Storage class does not exist
- Resolving an issue where the primary server or media server deployment does not proceed
- Resolving an issue of failed probes
- Resolving token issues
- Resolving an issue related to insufficient storage
- Resolving an issue related to invalid nodepool
- Resolving a token expiry issue
- Resolve an issue related to KMS database
- Resolve an issue related to pulling an image from the container registry
- Resolving an issue related to recovery of data
- Check primary server status
- Pod status field shows as pending
- Ensure that the container is running the patched image
- Getting EEB information from an image, a running container, or persistent data
- Resolving the certificate error issue in NetBackup operator pod logs
- Pod restart failure due to liveness probe time-out
- NetBackup messaging queue broker take more time to start
- Host mapping conflict in NetBackup
- Issue with capacity licensing reporting which takes longer time
- Local connection is getting treated as insecure connection
- Primary pod is in pending state for a long duration
- Backing up data from Primary server's /mnt/nbdata/ directory fails with primary server as a client
- Storage server not supporting Instant Access capability on Web UI after upgrading NetBackup
- Taint, Toleration, and Node affinity related issues in cpServer
- Operations performed on cpServer in environment.yaml file are not reflected
- Elastic media server related issues
- Failed to register Snapshot Manager with NetBackup
- Post Kubernetes cluster restart, flexsnap-listener pod went into CrashLoopBackoff state or pods were unable to connect to flexsnap-rabbitmq
- Post Kubernetes cluster restart, issues observed in case of containerized Postgres deployment
- Request router logs
- Issues with NBPEM/NBJM
- Issues with logging feature for Cloud Scale
- The flexsnap-listener pod is unable to communicate with RabbitMQ
- Troubleshooting AKS-specific issues
- Troubleshooting EKS-specific issues
- Troubleshooting issue for bootstrapper pod
- Troubleshooting AKS and EKS issues
- Appendix A. CR template
- Appendix B. MSDP Scaleout
- About MSDP Scaleout
- Prerequisites for MSDP Scaleout (AKS\EKS)
- Limitations in MSDP Scaleout
- MSDP Scaleout configuration
- Installing the docker images and binaries for MSDP Scaleout (without environment operators or Helm charts)
- Deploying MSDP Scaleout
- Managing MSDP Scaleout
- MSDP Scaleout maintenance
Patching of containers
This section describes the procedure for patching of the following containers (listed with examples):
Primary (main) containers
For example,
netbackup/main:10.5-patchMedia containers
For example,
netbackup/media:10.5-patchPoddependency-init containers
For example,
netbackup/operator:10.5-patchFluentbit collector pods
For example,
netbackup/fluentbit: 10.5-patchFluentbit cleanup container
For example,
netbackup/fluentbit-log-cleanup: 10.5-patchPostgreSQL pods
Table: Primary and Media container keywords
Profile name | Main container keyword | Init container keyword | Sidecar container keyword |
|---|---|---|---|
Primary container keywords | |||
Primary |
Example,
| ||
Nbatd |
Example,
|
Example,
|
Example,
|
Mqbroker |
Example,
|
Example,
| |
Webservice |
Example,
|
Example,
|
Example,
Example,
|
Policyjob |
Example,
|
Example,
policyjob.secure-comms Example,
policyjob.nbhousekeeping Example,
| |
Policyjobmgr |
Example,
|
Example,
Example,
Example,
| |
Requestrouter |
Example,
| ||
Bootstrapper |
Example,
| ||
Media container keywords | |||
Media |
|
|
|
To patch the main, media, poddependency-init container
- Use the following command to obtain the environment name:
$ kubectl get environments -n <namespace>
- Depending on the container, use the following respective command to verify if serviceImageTag is present:
For primary container:
$ kubectl get environment -n <namespace> -o=jsonpath='{$.items[0].spec.primary.serviceImageTag}'
For media container:
$ kubectl get environment -n <namespace> -o=jsonpath='{$.items[0].spec.mediaServers[0].serviceImageTag}'
For poddependency-init container:
$ kubectl get environment -n <namespace> -o=jsonpath='{$.items[0].spec.mediaServers[0].serviceImageTag}'
Depending on the output perform one of the following:
If serviceImageTag is present, then run the following command by changing the value field. For example, change
10.5-patchto the required image tag:For primary container:
$ kubectl patch environment <env-name> -n <namespace> --type=json --patch '[{"op": "replace", "path": "/spec/primary/serviceImageTag/primary.main", "value": "10.5-patch"}]'
For patching init containers of primary server:
kubectl patch environment <env-name> -n <namespace> --type=json --patch '[{"op": "replace", "path": "/spec/primary/serviceImageTag/<init container keyword>", "value": "10.5-patch"}]'
For patching sidecar container of primary server
kubectl patch environment <env-name> -n <namespace> --type=json --patch '[{"op": "replace", "path": "/spec/primary/serviceImageTag/<SideCar containers keyword>", "value": "10.5-patch"}]'
For media container:
$ kubectl patch environment <env-name> -n <namespace> --type=json --patch '[{"op": "replace", "path": "/spec/mediaServers/0/serviceImageTag/media.main", "value": "10.5-patch"}]'
For patching init containers of media server:
kubectl patch environment <env-name> -n <namespace> --type=json --patch '[{"op": "replace", "path": "/spec/mediaServers/0/serviceImageTag/<Init containers keyword>", "value": "10.5-patch"}]'
For patching sidecar container of media server
kubectl patch environment <env-name> -n <namespace> --type=json --patch '[{"op": "replace", "path": "/spec/mediaServers/0/serviceImageTag/<container keyword>", "value": "10.5-patch"}]'
For poddependency-init container:
$ kubectl patch environment <env-name> -n <namespace> --type=json --patch '[{"op": "replace", "path": "/spec/mediaServers/0/serviceImageTag/media.pod-dependency-init", "value": "10.5-patch"}]'
Or
If serviceImageTag is not present, then run the following command by changing the value field. For example, change
10.5-patchto the required image tag:For primary container:
$ kubectl patch environment <env-name> -n <namespace> --type=json --patch '[{"op": "replace", "path": "/spec/primary/serviceImageTag", "value": {}},{"op": "replace", "path": "/spec/primary/serviceImageTag/primary.main", "value": "10.5-patch"}]'
For patching init containers of primary server:
kubectl patch environment <env-name> -n <namespace> --type=json --patch '[{"op": "replace", "path": "/spec/primary/serviceImageTag", "value": {}},{"op": "replace", "path": "/spec/primary/serviceImageTag/<init container keyword>", "value": "10.5-patch"}]'
For patching sidecar container of primary server
kubectl patch environment <env-name> -n <namespace> --type=json --patch '[{"op": "replace", "path": "/spec/primary/serviceImageTag", "value": {}},{"op": "replace", "path": "/spec/primary/serviceImageTag/<SideCar containers keyword>", "value": "10.5-patch"}]'
For media container:
$ kubectl patch environment nbu -n netbackup --type=json --patch '[{"op": "replace", "path": "/spec/mediaServers/0/serviceImageTag", "value": {}},{"op": "replace", "path": "/spec/mediaServers/0/serviceImageTag/media.main", "value": "10.5-patch"}]'
For patching init containers of media server:
kubectl patch environment <env-name> -n <namespace> --type=json --patch '[{"op": "replace", "path": "/spec/mediaServers/0/serviceImageTag", "value": {}},{"op": "replace", "path": "/spec/mediaServers/0/serviceImageTag/<Init containers keyword>", "value": "10.5-patch"}]'
For patching sidecar container of media server
kubectl patch environment <env-name> -n <namespace> --type=json --patch '[{"op": "replace", "path": "/spec/mediaServers/0/serviceImageTag", "value": {}},{"op": "replace", "path": "/spec/mediaServers/0/serviceImageTag/<SideCar containers keyword>", "value": "10.5-patch"}]'
For poddependency-init container:
$ kubectl patch environment <env-name> -n <namespace> --type=json --patch '[{"op": "replace", "path": "/spec/mediaServers/0/serviceImageTag", "value": {}},{"op": "replace", "path": "/spec/mediaServers/0/serviceImageTag/media.pod-dependency-init", "value": "10.5-patch"}]'
Note:
For patching the sidecar containers for primary server:
If a specific container is patched with some image(ex. nbatd.pbx: 10.5-patch) and you want to patch that container universally (ex. pbx: 10.5-patch1), then you have to first delete the previous entry for that container from the environment.
From version 10.5, you can also patch the fluentbit collector pods using the fluentbit containers. The fluentbit collector, daemonset, and sidecar are the same container images.
Fluentbit container patch image example: netbackup/fluentbit:10.5-patch
Fluentbit cleanup container patch image example: netbackup/fluentbit-log-cleanup:10.5-patch
To patch the fluentbit collector pods
- Using the following command, get the currently configured values. Save the values in a file.
helm get values -n <namespace> fluentbit | grep -v "USER-SUPPLIED VALUES:" > <values file name>
- Update the output file to reflect new image tags:
Edit and update the tags in below json path file for the fluentbit-collector and execute the command:
fluentbit.image.tag to 10.5-patch (or actual patch tag)
Edit and update the tags in below json path file for the fluenbit-log-cleanup image and execute the command:
fluentbit.cleanup.image.tag to 10.5-patch (or actual patch tag)
- To update the deployment, run the helm command:
helm upgrade --install -n <namespace> fluentbit <fluentbit tgz> -f <values file name>
Note:
The tarball you need for this command is the one containing the helm templates, not the docker images. It is usually named fluentbit-<version>.tgz
The procedure describes the steps to update the image for NetBackup PostgreSQL pod in 10.5.
To update the containerized PostgreSQL pod for NetBackup version 10.5
Consider a scenario - In 10.5-xxxx deployment with netbackup-postgresql version 16.10.x.x, upgrade to netbackup-postgresql version 16.10.x.x using tag 10.5-patch2
- Suspend the backup job processing using the command:
nbpemreq -suspend_scheduling
- Load, tag, and push the new NetBackup PostgreSQL image using the commands:
To load the PostgreSQL imagedocker load -i VRTSnetbackup-postgresql-16.4.0002.tar.gz
To load the PostgreSQL-upgrade imagesdocker load -i VRTSnetbackup-postgresql-upgrade-16.4-0002.tar.gz
To tag the new imagesdocker tag netbackup/postgresql:16.4-0002 exampleacr.azurecr.io/netbackup/postgresql:10.5-patch2
docker tag localhost/netbackup/postgresql-upgrade:16.4-0002 exampleacr.azurecr.io/netbackup/postgresql-upgrade:10.5_patch2
To push the PostgreSQL imagedocker push exampleacr.azurecr.io/netbackup/postgresql:10.5-patch2
docker push exampleacr.azurecr.io/netbackup/postgresql-upgrade:10.5-patch2
- Upgrade the PostgreSQL using Helm chart:
To save PostgreSQL chart values to a file: helm show values postgresql-10.5-64.tgz > postgres-values.yaml
vi postgres-values.yaml
Upgrade the PostgreSQL: helm upgrade --install postgresql postgresql-10.5-64.tgz -f postgres-values.yaml -n netbackup
- Resume the backup job processing using the command:
nbpemreq -resume_scheduling