NetBackup™ Deployment Guide for Kubernetes Clusters
- Introduction
- Section I. Deployment
- Prerequisites for Kubernetes cluster configuration
- Deployment with environment operators
- Deploying NetBackup
- Preparing the environment for NetBackup installation on Kubernetes cluster
- Recommendations of NetBackup deployment on Kubernetes cluster
- Limitations of NetBackup deployment on Kubernetes cluster
- Primary and media server CR
- Configuring NetBackup IT Analytics for NetBackup deployment
- Managing NetBackup deployment using VxUpdate
- Migrating the cloud node for primary or media servers
- Deploying NetBackup using Helm charts
- Deploying MSDP Scaleout
- Deploying MSDP Scaleout
- Prerequisites for AKS
- Prerequisites for EKS
- Installing the docker images and binaries
- Initializing the MSDP operator
- Configuring MSDP Scaleout
- Using MSDP Scaleout as a single storage pool in NetBackup
- Configuring the MSDP cloud in MSDP Scaleout
- Using S3 service in MSDP Scaleout for AKS
- Enabling MSDP S3 service after MSDP Scaleout is deployed for AKS
- Deploying Snapshot Manager
- Verifying Cloud Scale deployment
- Section II. Monitoring and Management
- Monitoring NetBackup
- Monitoring MSDP Scaleout
- Monitoring Snapshot Manager
- Managing the Load Balancer service
- Managing MSDP Scaleout
- Managing PostrgreSQL DBaaS
- Performing catalog backup and recovery
- Setting key parameters in Cloud Scale deployments
- Section III. Maintenance
- MSDP Scaleout Maintenance
- PostgreSQL DBaaS Maintenance
- Upgrading
- Uninstalling
- Troubleshooting
- Troubleshooting AKS and EKS issues
- View the list of operator resources
- View the list of product resources
- View operator logs
- View primary logs
- Socket connection failure
- Resolving an issue where external IP address is not assigned to a NetBackup server's load balancer services
- Resolving the issue where the NetBackup server pod is not scheduled for long time
- Resolving an issue where the Storage class does not exist
- Resolving an issue where the primary server or media server deployment does not proceed
- Resolving an issue of failed probes
- Resolving token issues
- Resolving an issue related to insufficient storage
- Resolving an issue related to invalid nodepool
- Resolving a token expiry issue
- Resolve an issue related to KMS database
- Resolve an issue related to pulling an image from the container registry
- Resolving an issue related to recovery of data
- Check primary server status
- Pod status field shows as pending
- Ensure that the container is running the patched image
- Getting EEB information from an image, a running container, or persistent data
- Resolving the certificate error issue in NetBackup operator pod logs
- Pod restart failure due to liveness probe time-out
- NetBackup messaging queue broker take more time to start
- Host mapping conflict in NetBackup
- Issue with capacity licensing reporting which takes longer time
- Local connection is getting treated as insecure connection
- Primary pod is in pending state for a long duration
- Backing up data from Primary server's /mnt/nbdata/ directory fails with primary server as a client
- Storage server not supporting Instant Access capability on Web UI after upgrading NetBackup
- Taint, Toleration, and Node affinity related issues in cpServer
- Operations performed on cpServer in environment.yaml file are not reflected
- Elastic media server related issues
- Failed to register Snapshot Manager with NetBackup
- Pods unable to connect to flexsnap-rabbitmq post Kubernetes cluster restart
- Troubleshooting AKS-specific issues
- Troubleshooting EKS-specific issues
- Troubleshooting AKS and EKS issues
- Appendix A. CR template
Changing database server password in DBaaS
AKS-specific
- Launch an Azure CLI pod into the AKS cluster using the following command:
$ kubectl run az-cli --image=mcr.microsoft.com/azure-cli --command sleep infinity
Note:
Access to Azure Key Vault is restricted to specific subnets. Passwords stored in Azure Key Vault can be easily updated from a pod running in AKS.
- Exec into the Azure CLI pod as follows:
$ kubectl exec -it az-cli -- /bin/ash
- From Azure CLI pod, log into Azure account:
$ az login --scope https://graph.microsoft.com//.default
- (Optional) Create a key vault policy to allow the current user to retrieve the database credential.
Obtain the name of your resource group, key vault and ID of the current user by using the following respective commands:
Resource group name:
$ RESOURCE_GROUP=<resource_group_name>
Key vault name:
$ KEY_VAULT_NAME=$(az keyvault list --resource-group $RESOURCE_GROUP --resource-type vault | jq -r '.[].name')
Current user ID name:
$ USER_ID=$(az account show | jq -r '.user.name')
Create a key vault access policy as follows:
$ az keyvault set-policy -n $KEY_VAULT_NAME --upn $USER_ID --resource-group $RESOURCE_GROUP --secret-permissions all
- Obtain the login name for the key vault (DBADMINUSER):
$ DBADMINUSER=$(az keyvault secret show --vault-name $KEY_VAULT_NAME --name dbadminlogin | jq -r .value)
- Obtain the password for the key vault (OLD_DBADMINPASSWORD):
$ OLD_DBADMINPASSWORD=$(az keyvault secret show --vault-name $KEY_VAULT_NAME --name dbadminpassword | jq -r .value)
- Set the new password as follows:
Before setting the new password ensure that you know your database server name or obtain it by using the following command:
$ DBSERVER=$(az postgres flexible-server list --resource-group $RESOURCE_GROUP | jq -r '.[].name')
Use the following command to set the new password:
$ az postgres flexible-server execute -p $OLD_DBADMINPASSWORD -u $DBADMINUSER -n $DBSERVER -d postgres -q "ALTER USER\"nbdbadmin\" WITH PASSWORD '<new_password>';"
Note:
To install
rdbms-connectextension, ensure that you select the Y option. - Use the following command to verify if the password is using the correct encryption method (SCRAM-SHA-256):
$ az postgres flexible-server execute -p "<new_password>" -u $DBADMINUSER -n $DBSERVER -d postgres -q "SELECT * from azure_roles_authtype();"
+---------------------------+-----------+ | rolename | authtype | |---------------------------+-----------| | azuresu | NOLOGIN | | pg_database_owner | NOLOGIN | | pg_read_all_data | NOLOGIN | | pg_write_all_data | NOLOGIN | | pg_monitor | NOLOGIN | | pg_read_all_settings | NOLOGIN | | pg_read_all_stats | NOLOGIN | | pg_stat_scan_tables | NOLOGIN | | pg_read_server_files | NOLOGIN | | pg_write_server_files | NOLOGIN | | pg_execute_server_program | NOLOGIN | | pg_signal_backend | NOLOGIN | | azure_pg_admin | NOLOGIN | | replication | NOLOGIN | | nbdbadmin | SCRAM-256 | +---------------------------+-----------+ SELECT 15 Time: 0.009s
- Store the updated password in key vault:
$ az keyvault secret set --vault-name $KEY_VAULT_NAME --name dbadminpassword --value "<new_password>"
- (Optional) Delete the key vault access policy created in step 4 above:
$ az keyvault delete-policy -n $KEYVAULT --upn $USER_ID
- Exit from the azure CLI pod:
$ exit
- Delete the az CLI pod:
$ kubectl delete pod az-cli
- (Applicable only for an existing cloudscale deployment) Restart the primary pod:
$ kubectl rollout restart "statefulset/${PRIMARY}" --namespace "${NAMESPACE}"
In the above command,
NAMESPACE is the namespace containing your NetBackup deployment
PRIMARY is the name of primary pod's stateful set
Use the following command to obtain NAMESPACE and PRIMARY:
$ kubectl get --namespace "${NAMESPACE}" primaryserver -o jsonpath='{.items[0].status.attributes.resourceName}'
EKS-specific
- Use lambda function to change the password.
LAMBDA_ARN is the ARN of the password changing lambda function. This can be obtained from the lambda function page on AWS console.
NEW_PASSWORD is the new password to be used.
$ aws lambda invoke --function-name $LAMBDA_ARN \ --cli-binary-format raw-in-base64-out --payload '{"password":"$NEW_PASSWORD"}' \ response_file
- Wait for database to be available.
Obtain the POSTGRESQL_ID (database identifier)of your RDS Postgres database from the RDS database page of the AWS console, using the following command:
$ aws rds wait db-instance-available --db-instance-identifier $POSTGRESQL_ID
- Restart the primary pod:
$ kubectl rollout restart "statefulset/${PRIMARY}" --namespace "${NAMESPACE}"
In the above command,
NAMESPACE is the namespace containing your NetBackup deployment
PRIMARY is the name of primary pod's stateful set
Use the following command to obtain NAMESPACE and PRIMARY:
$ kubectl get --namespace "${NAMESPACE}" primaryserver -o jsonpath='{.items[0].status.attributes.resourceName}'