NetBackup™ for Kubernetes Administrator's Guide
- Overview of NetBackup for Kubernetes
- Deploying and configuring the NetBackup Kubernetes operator
- Prerequisites for NetBackup Kubernetes Operator deployment
- Deploy service package on NetBackup Kubernetes operator
- Port requirements for Kubernetes operator deployment
- Upgrade the NetBackup Kubernetes operator
- Delete the NetBackup Kubernetes operator
- Configure NetBackup Kubernetes data mover
- Automated configuration of NetBackup protection for Kubernetes
- Customize Kubernetes workload
- Troubleshooting NetBackup servers with short names
- Data mover pod schedule mechanism support
- Validating accelerator storage class
- Deploying certificates on NetBackup Kubernetes operator
- Managing Kubernetes assets
- Managing Kubernetes intelligent groups
- Managing Kubernetes policies
- Protecting Kubernetes assets
- Managing image groups
- Protecting Rancher managed clusters in NetBackup
- Recovering Kubernetes assets
- About incremental backup and restore
- Enabling accelerator based backup
- Enabling FIPS mode in Kubernetes
- About Openshift Virtualization support
- Troubleshooting Kubernetes issues
- Error during the primary server upgrade: NBCheck fails
- Error during an old image restore: Operation fails
- Error during persistent volume recovery API
- Error during restore: Final job status shows partial failure
- Error during restore on the same namespace
- Datamover pods exceed the Kubernetes resource limit
- Error during restore: Job fails on the highly loaded cluster
- Custom Kubernetes role created for specific clusters cannot view the jobs
- Openshift creates blank non-selected PVCs while restoring applications installed from OperatorHub
- NetBackup Kubernetes operator become unresponsive if PID limit exceeds on the Kubernetes node
- Failure during edit cluster in NetBackup Kubernetes 10.1
- Backup or restore fails for large sized PVC
- Restore of namespace file mode PVCs to different file system partially fails
- Restore from backup copy fails with image inconsistency error
- Connectivity checks between NetBackup primary, media, and Kubernetes servers.
- Error during accelerator backup when there is no space available for track log
- Error during accelerator backup due to track log PVC creation failure
- Error during accelerator backup due to invalid accelerator storage class
- Error occurred during track log pod start
- Failed to setup the data mover instance for track log PVC operation
- Error to read track log storage class from configmap
NetBackup Kubernetes operator become unresponsive if PID limit exceeds on the Kubernetes node
In Linux systems, there is an initd or system process running as PID 1 to reap zombie processes. Containers that do not have such an initd process would keep spawning zombie processes.
After certain time period these zombie processes accumulates and then reaches the max limit of PIDs set on the Kubernetes node.
In NetBackup Kubernetes operator, nbcertcmdtool spawns child processes to carry out certificate-related operations. On completion of the operation, the processes get orphan and are not reaped. Eventually it hits the max PID limit and NetBackup Kubernetes operator becomes unresponsive.
Error message: login pod/nbukops-controller-manager-67f5498bbb-gn9zw -c netbackupkops -n nbukops ERRO[0005] exec failed: container_linux.go:380: starting container process caused: read init-p: connection reset by peer a command that is terminated with exit code 1.
Recommended actions:
To fix the PID limit exceed issue, you can use the Initd script. Initd script acts as parent process or entry point script to the controller pod.
As a parent process it attaches zombie process to itself after the child process completion to terminate the persistent zombie process. It also helps you to shut down the container gracefully. Initd script is available in NBUKOPs build version 10.0.1.
Use the following steps to remove the existing nbcertcmdtool zombie processes:
Describe the NetBackup operator pod and find the Kubernetes node on which the controller pod is running. Run the command:
kubectl describe -c netbackupkops <NB k8s operator pod name> -n <namespace>
Log on to the Kubernetes node, run the command:
kubectl debug node/nodename
Terminate the nbcertcmdtool zombie processes, run the command:
ps -ef | grep "\[nbcertcmdtool\] <defunct>" | awk '{print $3}' | xargs kill -9
Note:
These steps terminate all the zombie processes for that worker node. But it resolves the issue temporarily. For a permanent solution, you must deploy a new KOps build with Initd script.