NetBackup™ Deployment Guide for Kubernetes Clusters
- Introduction
- Section I. Configurations
- Prerequisites
- Recommendations and Limitations
- Configurations
- Configuration of key parameters in Cloud Scale deployments
- Section II. Deployment
- Section III. Monitoring and Management
- Monitoring NetBackup
- Monitoring Snapshot Manager
- Monitoring MSDP Scaleout
- Managing NetBackup
- Managing the Load Balancer service
- Managing PostrgreSQL DBaaS
- Performing catalog backup and recovery
- Managing MSDP Scaleout
- Section IV. Maintenance
- MSDP Scaleout Maintenance
- PostgreSQL DBaaS Maintenance
- Patching mechanism for Primary and Media servers
- Upgrading
- Cloud Scale Disaster Recovery
- Uninstalling
- Troubleshooting
- Troubleshooting AKS and EKS issues
- View the list of operator resources
- View the list of product resources
- View operator logs
- View primary logs
- Socket connection failure
- Resolving an issue where external IP address is not assigned to a NetBackup server's load balancer services
- Resolving the issue where the NetBackup server pod is not scheduled for long time
- Resolving an issue where the Storage class does not exist
- Resolving an issue where the primary server or media server deployment does not proceed
- Resolving an issue of failed probes
- Resolving token issues
- Resolving an issue related to insufficient storage
- Resolving an issue related to invalid nodepool
- Resolving a token expiry issue
- Resolve an issue related to KMS database
- Resolve an issue related to pulling an image from the container registry
- Resolving an issue related to recovery of data
- Check primary server status
- Pod status field shows as pending
- Ensure that the container is running the patched image
- Getting EEB information from an image, a running container, or persistent data
- Resolving the certificate error issue in NetBackup operator pod logs
- Pod restart failure due to liveness probe time-out
- NetBackup messaging queue broker take more time to start
- Host mapping conflict in NetBackup
- Issue with capacity licensing reporting which takes longer time
- Local connection is getting treated as insecure connection
- Primary pod is in pending state for a long duration
- Backing up data from Primary server's /mnt/nbdata/ directory fails with primary server as a client
- Storage server not supporting Instant Access capability on Web UI after upgrading NetBackup
- Taint, Toleration, and Node affinity related issues in cpServer
- Operations performed on cpServer in environment.yaml file are not reflected
- Elastic media server related issues
- Failed to register Snapshot Manager with NetBackup
- Post Kubernetes cluster restart, flexsnap-listener pod went into CrashLoopBackoff state or pods were unable to connect to flexsnap-rabbitmq
- Post Kubernetes cluster restart, issues observed in case of containerized Postgres deployment
- Troubleshooting AKS-specific issues
- Troubleshooting EKS-specific issues
- Troubleshooting AKS and EKS issues
- Appendix A. CR template
Enable request logging, update configuration, and copying files from/to PostgreSQL pod
This section provides the steps to enable, update and copy the container based PostgreSQL deployment for NetBackup database service.
Following are the PostgreSQL log and configuration files location:
PostgreSQL log:
/netbackup/postgresqldb/logs/*PostgreSQL config:
/netbackup/postgresqldb/postgresql.confPostgreSQL hba:
/netbackup/postgresqldb/pg_hba.confPostgreSQL bin:
/usr/openv/db/binPgBouncer ini:
/home/nbsvcusr/postgresqldb/pgbouncer.ini
Perform the following to update the configuration or enable request for logging when there is a separate PostgreSQL container:
Run the following command to exec into the Postgres pod:
kubectl exec -it <postgres-pod-name> -n netbackup - - bash
Run the following command to list the PostgreSQL logs:
ls -l /netbackup/postgresqldb/log/
Run the following command to view the conf file:
cat /netbackup/postgresqldb/posgresql.conf
Modify the PostgreSQL configurations, perform the following:
Note:
Any changes done to
/netbackup/postgresqldb/postgresql.conffile would be lost during the PostgreSQL pod restart.Method 1
Run the following command to exec into the Postgres pod:
kubectl exec -it <postgres-pod-name> -n netbackup - - bash
Set the value of parameter to 0 as follows:
sed -i 's/^#log_parameter_max_length = -1/log_parameter_max_length = 0/' /netbackup/postgresqldb/postgresql.conf
Restart Postgres as follows:
su nbsvcusr -c 'pg_ctl stop -w -D /netbackup/postgresqldb'
su nbsvcusr -c 'pg_ctl start -w -D /netbackup/postgresqldb'
Method 2 (Requires access to the internet)
Run the following command to exec into the Postgres pod:
kubectl exec -it nb-postgresql-0 -n netbackup - - bash
Run the following command to download the
vi:microdnf install vi
Edit the
postgresql.conffile as follows:vi /netbackup/postgresqldb/postgresql.conf
Restart Postgres as follows:
su nbsvcusr -c 'pg_ctl stop -w -D /netbackup/postgresqldb'
su nbsvcusr -c 'pg_ctl start -w -D /netbackup/postgresqldb'
To deploy PostgreSQL with custom configuration
- Start PostgreSQL with custom configuration files as follows:
Create
psqlconfdir and copy the configuration files:mkdir psqlconf
kubectl cp netbackup/nb-postgresql-0:/netbackup/postgresqldb/postgresql.conf psqlconf/postgresql.conf kubectl cp netbackup/nb-postgresql-0:/netbackup/postgresqldb/pg_hba.conf psqlconf/pg_hba.conf kubectl cp netbackup/nb-postgresql-0:/home/nbsvcusr/postgresqldb/pgbouncer.ini psqlconf/pgbouncer.ini
Modify the settings of postgresql.conf file:
vi psqlconf/postgresql.conf
Create the
nbsqlconfconfigMap:kubectl create configmap nbpsqlconf --from-file=psqlconf -n netbackup
Use the following command to save the PostgreSQL chart values to a file:
helm show values postgresql-<version>.tgz > postgres-values.yaml
Modify postgresql helm chart input file: :
vi postgres-values.yaml
Uncomment configMapName, mountConf, pgbouncerIniPath, postgresqlConfFilePath, pgHBAConfFilePath as in the following example:
# Default values for postgresql. global: environmentNamespace: "netbackup" containerRegistry: "123456789abc.dkr.ecr.us-east-1.amazonaws.com" postgresql: replicas: 1 # The values in the image (name, tag) are placeholders. These will be set image: name: "netbackup/postgresql" tag: "10.4" pullPolicy: Always service: serviceName: nb-postgresql volume: volumeClaimName: nb-psql-pvc volumeDefaultMode: 0640 pvcStorage: 5Gi configMapName: nbpsqlconf storageClassName: nb-disk-premium mountPathData: /netbackup/postgresqldb secretMountPath: /netbackup/postgresql/keys/server mountConf: /netbackup timezone: null securityContext: runAsUser: 0 createCerts: true pgbouncerIniPath: /netbackup/pgbouncer.ini serverSecretName: postgresql-server-crt clientSecretName: postgresql-client-crt dbSecretName: dbsecret dbPort: 13785 pgbouncerPort: 13787 dbAdminName: postgres initialDbAdminPassword: postgres dataDir: /netbackup/postgresqldb postgresqlConfFilePath: /netbackup/postgresql.conf pgHbaConfFilePath: /netbackup/pg_hba.conf defaultPostgresqlHostName: nb-postgresqlReinstall postgresql helm chart using the modified input file:
helm upgrade --install postgresql postgresql-10.4.1.tgz -f postgres-values.yaml -n netbackup
Wait for the postgres pod to restart.
To enable request logging
- Enable request logging by performing the following changes to
postgresql.conffile:log_min_duration_statement = 0
Note:
The value 0 means the setting is the threshold in milliseconds after which the statement is to be logged.
log_checkpoints = on log_connections = on log_disconnections = on log_lock_waits = on log_temp_files = 0 log_autovacuum_min_duration = 0 log_error_verbosity = default
Run the following command to exec into the Postgres pod:
kubectl exec -it <postgres-pod-name> -n netbackup - - bash
Run the following command to list the PostgreSQL logs:
ls -l /netbackup/postgresqldb/log/
Run the following command to view the conf file:
cat /netbackup/postgresqldb/posgresql.conf
Perform one of the following methods to copy files out of PostgreSQL container:
Method 1:
Run the following commands:
kubectl exec nb-postgresql-0 -n netbackup -- cat /netbackup/postgresqldb/postgresql.conf > /tmp/postgresql.conf
kubectl exec nb-postgresql-0 -n netbackup -- cat /netbackup/postgresqldb/log/postgresql-Tue.log > /tmp/postgresql-Tue.log
Method 2 (Requires access to the internet):
Exec into Postgres pod and install tar:
kubectl exec -i nb-postgresql-0 -n netbackup - - microdnf install tar
Copy the files:
kubectl cp netbackup/nb-postgresql-0:/netbackup/postgresqldb/postgresql.conf /tmp/postgresql.conf