NetBackup CoPilot - Best Practices for setup and tuning CoPilot volume and NFS mount

Description

Customers implementing NetBackup CoPilot for Oracle are advised to use the following tuning values for vxfs and nfs to achieve the best possible performance.

NetBackup Appliance Media Server

1. Settings for the CoPilot NFS Share

When creating the NetBackup CoPilot share (Web UI or CLISH), use these settings:
(rw, no_root_squash, insecure)

2. VxFS Filesystem Tuning

After creating the CoPilot share, open an elevated command prompt on the Appliance.
Edit the VxFS file system tuning parameters file '/etc/vx/tunefstab'.
Add the following value to the end of the line for the CoPilot volume to increase the extent size.

"ftrunc_up_zfod_alloc=33554432"

Example Before

/dev/vx/dsk/nbuapp/dbvol_CoPilotShare     dalloc_enable=0,discovered_direct_iosz=256k,initial_extent_size=128,read_nstream=8,write_pref_io=1M

Example After

/dev/vx/dsk/nbuapp/dbvol_CoPilotShare     dalloc_enable=0,discovered_direct_iosz=256k,initial_extent_size=128,read_nstream=8,write_pref_io=1M,ftrunc_up_zfod_alloc=33554432

3. Increase NFS Server Threads

The default thread count is 32 on current NetBackup Appliance releases. The thread count value to specify depends on the I/O load pattern for the Oracle operations (OLAP or OLTP), the network speed of the clients, and how many concurrent operations are required. To determine the best value, apply the expected backup load to the Appliance and monitor the load on nfsd. 

The default setting can be adjusted by using one of the two files below, increase the number of threads for nfsd to desired thread number.

RHEL 7 (deprecated) - Over rides nfs.conf below if set.
/etc/sysconfig/nfs
RPCNFSDCOUNT=32 

RHEL 7 (current) and RHEL 8
/etc/nfs.conf
[nfsd]
threads=32

Restart NFSD service to apply the tuning changes.
systemctl restart nfs

Example: How to Monitor NFS during a Backup

In the output below, monitor for the total load and thread usage during a backup. If TIME is near 100% for all threads, consider increasing the thread count. Restart NFS and repeat the process to find best thread count for the target environment.

maintenance-!> ps alx | egrep 'PID|nfsd'
F   UID   PID  PPID PRI  NI    VSZ   RSS WCHAN  STAT TTY        TIME COMMAND
0  1001  1068 17959  20   0 112820   964 pipe_w S+   pts/10     0:00 grep -E PID|nfsd
1     0  4990     2   0 -20      0     0 rescue S<   ?          0:00 [nfsd4_callbacks]
1     0  5008     2  20   0      0     0 svc_re S    ?          0:00 [nfsd]
1     0  5009     2  20   0      0     0 svc_re S    ?          0:00 [nfsd]
1     0  5010     2  20   0      0     0 svc_re S    ?          0:00 [nfsd]
1     0  5011     2  20   0      0     0 svc_re S    ?          0:00 [nfsd]
1     0  5012     2  20   0      0     0 svc_re S    ?          0:00 [nfsd]
1     0  5013     2  20   0      0     0 svc_re S    ?          0:00 [nfsd]
1     0  5014     2  20   0      0     0 svc_re S    ?          0:00 [nfsd]
1     0  5016     2  20   0      0     0 svc_re S    ?          0:00 [nfsd]
1     0  5017     2  20   0      0     0 svc_re S    ?          0:00 [nfsd]
.... 
(Output truncated for clarity)


Oracle Clients

Set the following values for the CoPilot NFS share in /etc/fstab:

<appliance_name>:<Copilot_share_name> /<local_directory_path> nfs rw,bg,hard,nointr,rsize=1048576,wsize=1048576,tcp,vers=3,timeo=600

Setup and configure Oracle Direct NFS (dNFS) to maximize the performance of the CoPilot operations.
Oracle dNFS allows for better multi-streaming throughput during backup and incremental merge operations.

Use the following options:

NFS_OPT="-o rw,bg,hard,nointr,rsize=1048576,wsize=1048576,tcp,vers=3,timeo=600"
 

Instructions on how to configure Oracle dNFS are in the following documentation from Oracle.

Oracle Support Document 762374.1
Step by Step - Configure Direct NFS Client (DNFS) on Linux
https://support.oracle.com/epmos/faces/DocumentDisplay?id=762374.1

-- Note: Access to this article requires a login to Oracle Support